docs · harnessbench-cli 0.1

Quick start

Five steps from nothing to your first verdict. Needs Node 20, git, and Claude Code signed in.

1. Set it up in your repoWrites the config, detects your test command, installs the recording hooks, and checks Claude Code is signed in.

   npx harnessbench-cli init

2. Work as usualUse Claude Code in that repo for a day or two. Each task is saved automatically. See them with npx harnessbench-cli tasks.
3. Change your harness on a branchA rule in CLAUDE.md, a hook, a skill, an MCP server. Commit it.
4. Run the comparisonEvery saved task runs twice, old harness and new.

   npx harnessbench-cli run --base main

5. Read the verdictShip, hold or revert, with the line responsible. Open the dashboard for detail.

   npx harnessbench-cli ui

docs · start

How it works

Record, replay, grade, decide.

The key idea: same prompt, same starting code, only the harness differs. Whatever changes in the agent's behaviour is caused by your change.

docs · use

Recording tasks

Automatic once init has installed the hooks.

A task is one prompt you typed plus everything until the agent stopped. Follow-ups you send in the same session are saved too and become the answers the replay uses if the agent asks again. Slash commands, system notifications and prompts sent from inside a replay are ignored.

By hand

npx harnessbench-cli tasks add "add a retry helper to the fetch client"
npx harnessbench-cli tasks label <id> "flaky webhook fix"

docs · use

Running a comparison

--base and --head are git refs. Only the harness files differ between the two runs.

npx harnessbench-cli run --base main                # main vs your branch
npx harnessbench-cli run --base main --head feat/x
npx harnessbench-cli run --base main --task webhook  # ids containing "webhook"
npx harnessbench-cli run --base main --parallel 2
npx harnessbench-cli run --base main --grader none   # checks only, no grader
npx harnessbench-cli run --base main --agent fake    # dry run, no Claude calls

Worktrees live under .hb/wt/ during the run and are removed after. --keep-worktrees keeps them. A run costs about what the tasks cost the first time, doubled. Start with --task on one task.

docs · use

Reading the result

A verdict, the reason, and the line that caused it.

Each task gets a score out of 10 for both runs. Under half a point of movement counts as unchanged. Flags appear when tokens or time exceed your budgets.

docs · use

Dashboard

A local page that reads your reports.

npx harnessbench-cli ui               # http://127.0.0.1:4321
npx harnessbench-cli ui --port 5000 --no-open

For each run: the verdict, a one-line explanation of how the agent behaved differently, which axes moved, and the tasks. Expand a task for the two runs side by side, the questions it asked, and the transcript. The harness diff is at the bottom.

docs · use

In CI

ci is run plus a markdown summary and an exit code.

on:
  pull_request:
    paths: [".claude/**", "CLAUDE.md", "AGENTS.md", ".mcp.json"]
jobs:
  regression:
    runs-on: self-hosted   # claude installed and signed in
    steps:
      - uses: actions/checkout@v4
        with: { fetch-depth: 0 }
      - run: npx harnessbench-cli ci --base origin/${{ github.base_ref }}

Exits 1 on hold or revert. --fail-on revert lets holds through. Writes to $GITHUB_STEP_SUMMARY when present. The recorded tasks must be available on the runner.

docs · use · team plan

Hosted dashboard

Reports your whole team can open, run history across branches, and a shared corpus of recorded tasks.

npx harnessbench-cli push              # upload the latest report
npx harnessbench-cli run --base main --push
npx harnessbench-cli tasks push        # share this repo's recorded tasks
npx harnessbench-cli pull              # fetch tasks teammates recorded
npx harnessbench-cli cloud --open      # open the dashboard, signed in

Reports are keyed by repository (the origin remote, e.g. acme/api), so everyone with a key from the same purchase sees the same runs. ci pushes automatically on the Team plan; set cloud.autoPush: false in the config to stop that, or --no-push for one run. The PR comment links to the hosted report.

What is uploaded

The report JSON, which includes scores, reasons, the harness diff and a condensed transcript excerpt for each run. Raw agent events and your source code never leave the machine. tasks push uploads prompts and transcripts, since teammates need those to replay.

Signing in on the web

harnessbench cloud --open opens the dashboard already signed in. On another device, paste your license key and the license id shown by harnessbench whoami. Nothing is stored server-side beyond a hash of the key.

docs · configure

Config file

.harnessbench/config.yml, written by init with comments.

agent:
  command: claude              # claude | fake | your own adapter
  permissionMode: acceptEdits  # see Permissions
  timeoutMinutes: 12
  parallel: 4
harnessPaths: [CLAUDE.md, AGENTS.md, .claude, .mcp.json]
signals:
  - name: tests
    command: npm test --silent
replay:
  policy: replay-then-proceed  # proceed | stop
cloud:
  url: https://api.harnessbench.run
  autoPush: false               # ci pushes regardless unless --no-push
grader:
  enabled: true
  model: claude-opus-5
rubric:
  axes: [...]
  hardFail: ["rm -rf /", "git push --force", "DROP TABLE"]
  budget: { tokensPct: 25, wallPct: 50 }
  verdict: { holdIfRegressionGt: 2.0, revertIfRegressionsGte: 3 }

docs · configure

Grading

Picked automatically, in this order.

1. API key present: the Anthropic API with grader.model and schema-enforced JSON.
2. Claude Code signed in: the same prompts through claude -p on your login.
3. Neither: checks only. Correctness from your tests, safety from commands run, scope from files touched.

Override with --grader none or --grader claude-sonnet-5. Every score comes with the grader's one-line reason.

docs · configure

Permissions

What the agent may do in the throwaway worktree without asking.

docs · configure

All commands

docs · more

Plans and license keys

Free for one person. A key unlocks the team features.

Buying a key

Team, per seat · Team, per repo · Business. Checkout is handled by Dodo Payments; cancel any time from the link in your receipt.

Activating a key

The key arrives by email after purchase. On each machine that should use it:

npx harnessbench-cli login HB-XXXX-XXXX-XXXX
npx harnessbench-cli whoami        # shows the active plan
npx harnessbench-cli logout        # frees the seat for another machine

Per-seat keys activate on one machine at a time. Per-repo and Business keys have no activation limit. In CI, set HARNESSBENCH_KEY in the environment instead of running login.

Keys are checked once a day and keep working for a week without network. Cancelling disables the key at the end of the period and the CLI drops back to Free. Nothing else changes.

docs · more

Troubleshooting

"Claude Code is not signed in"

Run claude once and sign in.

Tasks aren't recorded

Hooks load at session start; open a new Claude Code session after init. Check for three record --hook entries in .claude/settings.json or settings.local.json. HARNESSBENCH_DEBUG=1 prints hook decisions.

Both runs fail my checks

Worktrees start without dependencies. Node projects get node_modules linked in; for others, make your check command install first.

Where things live

.harnessbench/config.yml   settings and rubric
.harnessbench/tasks/       task.json + transcript.jsonl
.harnessbench/reports/     one JSON per run
.harnessbench/runs/        raw agent events        (gitignored)
.hb/wt/                    worktrees during a run  (gitignored)

Stop recording

Remove the record --hook entries from your Claude settings. Delete .harnessbench/ for everything else.