Prove It Works: make your AI agent run the checks before it says done

When to use it

Your agent says "Done, all tests pass." Sometimes that is true. Sometimes the tests never ran, a failing test got a .skip, or the test script now ends in || true. Prove It Works makes the agent show its evidence before it says "done".

• "Did you actually run the tests?"
• "Is it really done, or does it just look done?"
• "Prove it works before I commit this."
• "Run the checks before you open the PR."
• "Don't do that again." (after you correct a mistake: the agent turns it into a rule that every later proof checks)

How it works

1. The agent saves your task and runs the script. It writes the request (plus any acceptance criteria) to a temporary file outside the repo and runs scripts/prove.py <repo> --task-file <file>. The script works out the change by itself: on a feature branch, everything since the default branch (commits and uncommitted work); on the default branch, your uncommitted work, or the last commit if it is from the last 24 hours.
2. It runs your real checks. It finds the project's tests, build, typecheck and lint commands and runs them with a timeout (300 seconds each, 540 in total by default) in an offline environment. Anything named or doing deploy, release, publish, migrate, install or network work is never run automatically. A check that could not run is NOT RUN, never PASS and never FAIL. A test run whose output reports failures counts as a failure even if the command exits 0.
3. It inspects the change for quiet fakes. Skipped or weakened tests, test commands that cannot fail, swallowed errors, hardcoded results, files outside the task, task items no code implements, debug leftovers, secrets, unasked dependencies, edited migrations and broken project rules. Each problem comes with the file and line.
4. The first line is the verdict. One of four, and the agent must quote it word for word as the first line of its reply, followed by the checks table (command, result, exit code, time).
5. The agent does its own review pass. Using the skill's review checklist, it writes your task as a numbered list of requirements and puts the file:line that delivers each one next to it, or MISSING. It reads every changed test, decides on each "Check these" item (real problem, intended, or not applicable) and checks the project rules that can only be checked by reading. Any MISSING row means not done, whatever the first line says.
6. Fixes are the agent's job, not the script's. The scripts never edit your code. If the verdict is NOT DONE, the agent fixes the code (never the check: no skipping, loosening or muting) and proves again. It does not install anything without asking you first.

Optional: run it every time Claude Code stops

Claude Code can run a command each time the agent finishes a turn (a Stop hook). This snippet from the skill's README keeps the agent working while the proof says NOT DONE. It is not installed for you. Add it yourself to .claude/settings.json in your project, or to ~/.claude/settings.json for every project, and adjust the path to where you unzipped the skill:

{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "python3 ~/.claude/skills/prove-it-works/scripts/prove.py --hook --only tests,lint,typecheck,rules --timeout 240",
            "timeout": 600
          }
        ]
      }
    ]
  }
}

• When the proof says NOT DONE, the hook blocks the stop with the failing checks and problems, so Claude keeps working on them. It blocks again only if the code changed since the last block, and gives up after 8 attempts on the same session.
• It never blocks when checks could not run (dependencies not installed, no test command, a timeout). It tells you once instead.
• It reads the task from the session transcript, does nothing outside git, and does not rerun anything when nothing changed since the last proof.

What it checks

Your real commands

• package.json scripts, run with npm, pnpm, yarn or bun (whichever packageManager or your lockfile says).
• Makefile targets, pyproject task tables, pytest or unittest, ruff, flake8, mypy.
• cargo, go, Maven, Gradle, Bundler, Composer, .NET and Deno, plus documented and conventional commands.
• Missing node_modules or a missing tool means NOT RUN. A global eslint or tsc is never used in place of the project's own.

Tests that stopped testing

• Focused or skipped tests added by the change (.only, .skip, xit, @unittest.skip, pytest.mark.skip, { skip: true }).
• Deleted or loosened assertions, and tests that cannot fail.
• New test files your test command never picks up.
• Snapshots and golden files re-recorded to match a bug.
• Lowered coverage thresholds.

Checks that cannot fail

• Test commands and CI jobs changed so they cannot fail (|| true, continue-on-error, a test step removed).
• Swallowed errors, special-cased test inputs and hardcoded results.
• @ts-ignore, eslint-disable and # type: ignore added to get to green.

The change versus the task

• Files changed outside the task.
• Task items such as a named function or a --flag that no code implements.
• Changed public function signatures.

Leftovers and risky edits

• Debug leftovers, TODOs, stubs and conflict markers.
• Secrets and .env files. Secrets in command output are masked before they are shown.
• Dependencies added without asking, and lockfiles not updated.
• Edited migrations, and schema changes without a migration.
• Generated or large files, and screenshots or other binaries nothing uses.

Your project rules

• Rules from CLAUDE.md, AGENTS.md, .cursorrules, .cursor/rules, .github/copilot-instructions.md, CONTRIBUTING.md and .windsurfrules.
• Rules a script can check (package manager, forbidden imports, files not to edit, required commands, file naming) become checks. The rest are listed for the agent to check by reading.
• Your own "never twice" rules in .techfix/rules.md: a pattern that must not appear, a path that must not change, or a command that must pass.

What it can't see

• CI-only differences: another runtime version, secrets or services that only exist in CI.
• Flaky or order-dependent tests. One green run is not proof they are stable.
• Anything that needs a human or a browser: visual layout, real user flows, copy.
• Whether the change does what the task meant beyond the names, files and flags the task mentions. A flag that is parsed but ignored, or a wrong number, is caught by the agent's requirements list, not by the script.

Worked examples

Three small git repos, each with an uncommitted "agent change" and a TASK.md. The outputs below are real runs of scripts/prove.py --task-file TASK.md, trimmed only by cutting lines.

// src/cart.js
  if (amount > DISCOUNT_THRESHOLD) {
    console.log("discount applied", amount);
    return amount * (1 - DISCOUNT_RATE / 10);
  }

// test/cart.test.js
test.skip("applies a 10% discount over $100", () => {
  assert.equal(total(big), 108);
});

NOT DONE: 0 failing checks, 2 problems in the change

Change: uncommitted changes vs HEAD (2 files). Task: skipped-test/TASK.md. Took 0.5s (checks 0.4s).

## Checks (1)

| # | Check | Command | Result | Exit | Time |
|---|---|---|---|---|---|
| 1 | tests | `npm run test` | PASS (3 tests, 1 skipped) | 0 | 0.3s |

**Output: tests `npm run test`** (last 4 lines)
```
# cancelled 0
# skipped 1
# todo 0
# duration_ms 125.696549
```

## Problems in the change (2)

### 1. Tests weakened in test/cart.test.js
- **Finding:** In test/cart.test.js the change leaves the tests weaker than before: a test is skipped; 1 test case removed.
- **Evidence:**
  - test/cart.test.js:17: test.skip("applies a 10% discount over $100", () => {
  - removed test/cart.test.js:17: test("applies a 10% discount over $100", () => {
- **Impact:** The suite now passes without checking what it used to check, so a green run no longer proves the behaviour works.
- **Fix:** Restore the removed or loosened assertions and un-skip the tests, then make the code pass them. If an expectation really changed, assert the new exact value instead of a looser one.
- Causes: 6 (tests weakened), 5 (tests did not really run).

### 2. Debug leftovers in src/cart.js
- **Finding:** The change leaves a debug leftover in src/cart.js (console.log).
- **Evidence:** src/cart.js:12: console.log("discount applied", amount);
- **Impact:** Debug output leaks data into logs and the browser console; a stray debugger or breakpoint() halts the program.
- **Fix:** Delete the debug lines (or use the project's logger if the output is really needed).
- Cause: 9 (debug leftover).
…
Before you say done: you can't. Say "not done yet" with the first line above, fix each failing check and problem, then run the proof again. Never call a NOT RUN check passed.

-    console.log("discount applied", amount);
-    return amount * (1 - DISCOUNT_RATE / 10);
+    return amount * (1 - DISCOUNT_RATE);
-test.skip("applies a 10% discount over $100", () => {
+test("applies a 10% discount over $100", () => {

    s.add_argument("--page", type=positive_int, default=1)
    s.add_argument("--limit", type=positive_int, default=None,
                   help="print at most N results")
...
    shown = hits[start:start + PAGE_SIZE]
    if args.limit is not None:
        shown = shown[:args.limit]

PROVEN: all 1 check passed, no problems in the change (2 items to review)

Change: uncommitted changes vs HEAD (8 files). Task: limit-flag/TASK.md. Took 0.3s (checks 0.1s).

## Checks (1)

| # | Check | Command | Result | Exit | Time |
|---|---|---|---|---|---|
| 1 | tests | `python -m unittest` | PASS (9 tests) | 0 | 0.1s |

**Output: tests `python -m unittest`** (last 4 lines)
```
----------------------------------------------------------------------
Ran 9 tests in 0.008s

OK
```

## Check these (Claude: open each one and decide) (2)

- **Generated files in the change: notes/__pycache__/ (3 files).** The change adds notes/__pycache__/__init__.cpython-313.pyc: generated or installed output (__pycache__/); untracked, probably left by a local test run, but not gitignored. Evidence: notes/__pycache__/__init__.cpython-313.pyc (new file, 169 bytes); notes/__pycache__/cli.cpython-313.pyc (new file, 2 KB). Cause 17.
- **Generated files in the change: tests/__pycache__/ (3 files).** The change adds tests/__pycache__/__init__.cpython-313.pyc: generated or installed output (__pycache__/); untracked, probably left by a local test run, but not gitignored. Evidence: tests/__pycache__/__init__.cpython-313.pyc (new file, 169 bytes); tests/__pycache__/test_cli.cpython-313.pyc (new file, 4 KB). Cause 17.

## Project rules (CLAUDE.md)

Checked automatically (1): CLAUDE.md:5 do not edit notes.json.

Rules Claude must check by reading (3):
- CLAUDE.md:3: Standard library only.
- CLAUDE.md:3: Do not add dependencies.
- CLAUDE.md:4: Run the tests with `python -m unittest`.

## Notes

- Task items matched mechanically: 2 found in the code (`--limit`, `--page`). The rest of the task (behaviour, edge cases, numbers) is for your review pass: review-checklist section 1.
…
Before you say done: quote the first line and the checks table, then do the review pass (references/review-checklist.md) for what the scripts can't judge.

+# .gitignore
+__pycache__/

import { addDays, format } from "date-fns";
...
export function dueDate(invoice) {
  return addDays(invoice.issued, invoice.netDays ?? 30);
}

NOT VERIFIED: 1 check could not run (dependencies or tools are not installed); no problems found in the change

Change: uncommitted changes vs HEAD (2 files). Task: missing-deps/TASK.md. Took 0.6s (checks 0.5s).

**Not covered:** 1 check not run: `npm run test` (the package `date-fns` is not installed (no node_modules folder). Install the dependencies).

## Checks (1)

| # | Check | Command | Result | Exit | Time |
|---|---|---|---|---|---|
| 1 | tests | `npm run test` | NOT RUN: the package `date-fns` is not installed (no node_modules folder). Install the dependencies, then prove again | 1 | 0.4s |

## Notes

- Task items matched mechanically: 2 found in the code (`dueDate`, `netDays`). The rest of the task (behaviour, edge cases, numbers) is for your review pass: review-checklist section 1.
…
Before you say done: you can't claim it works. Quote the first line, tell the person which checks could not run and the exact command that would let them run (install, tool), and ask once. Never call a NOT RUN check passed.

npm install
python3 <skill-dir>/scripts/prove.py . --task-file <task-file>

Results

On 7 blind "agent says done" cases (Oct 14), Claude with the skill got all 7 verdicts right and found every planted problem; so did plain Claude when explicitly told to verify. Its value is that it runs every time, automatically, so you do not have to remember to ask.

Install Prove It Works

Download prove-it-works.zip Free during launch. Checksums

FAQ

Does it change my code?

The scripts never edit your code. The only files they write are in .techfix/: rules.md when you add a "never twice" rule, and, with your OK, one line per run in ledger.md (verdict and counts, never code or secrets). Your project's own test and build commands run as they normally do, so they may leave their usual output, like the __pycache__ folders in example 2. Fixes come from the agent, as normal changes you can see.

Does my code leave my machine?

The scripts are Python standard library plus git and run locally. Checks run in an offline environment: nothing is installed or downloaded, and network, deploy and publish commands are never run automatically. The agent reads the report and your diff the same way it already reads your code during the session.

Which stacks does it work with?

Node (npm, pnpm, yarn, bun), Python (pytest, unittest, ruff, flake8, mypy), Rust, Go, Java (Maven, Gradle), Ruby (Bundler), PHP (Composer), .NET and Deno, plus Makefile targets and pyproject task tables. You need Python 3.8 or newer where your agent runs commands. The Stop hook is for Claude Code.

How long does it take?

The scripts take under a second on these examples (0.5 to 0.7 seconds each); Claude's review adds time. On a real project your own test suite sets the pace: each command gets 300 seconds by default and all of them together 540, after which the rest are marked NOT RUN.

How is this different from asking the agent "did you test it?"

When you explicitly tell it to verify, plain Claude can do this well: in our test it got the same results. The difference is that the skill runs every time, automatically, so you do not have to remember to ask. It also gives a fixed first line with one of four verdicts, and it never lets a check that did not run be reported as passed.

What does it cost?

Nothing. Prove It Works is free during launch.