A two-part flowchart on a white background. The top row, labeled Build the benchmark, shows an existing task flowing into Benchproofer and then into a formal specification, a reference implementation, and a checked proof. The bottom row, labeled Evaluate in verified mode, shows a coding agent submitting a repository patch plus a Lean implementation and proof, which then pass three checks to reach a verified pass.
Flowchart from Amazon Science's 16 November 2026 post "BenchProofer: Formally verified coding benchmarks, on demand.", Amazon Science

要点

  • On 16 November 2026, the BenchProofer team at Amazon Science published "BenchProofer: Formally verified coding benchmarks, on demand." The open-source tool takes an existing coding task that already has a reference fix and tries to generate three artifacts: a formal specification, a precise, machine-checkable statement of what a correct answer must do; a reference implementation in a language suited to formal verification; and a proof, checked by a verifier, that the implementation meets the specification.
  • Alongside the tool, Amazon is releasing SWE-Proof, which adds those specifications, implementations, and proofs to all 500 tasks in SWE-Bench Verified, a widely used set of real GitHub bug-fix tasks used to score coding agents.
  • In testing, the audit flagged nearly one in three of Claude Opus 4.8's test-passing patches for behavior that differed from the reference fix. In a human-reviewed sample, about half of the flagged patches, 21 of 41, had real bugs.
  • When agents were given BenchProofer's specifications, verified pass rates rose: Claude Opus 4.8 from 84.6% to 93.6%, and GPT-5.5 from 79.6% to 94.4%. The post argues that this matters because benchmark scores are used to pick tools, and increasingly as a reward signal when companies train agents with reinforcement learning.
  • The method still depends on a known reference solution to build the specification, and on a human-reviewed sample to interpret the automated flags. Amazon's evaluations were run by the team that built the tool.

On 16 November 2026, Amazon Science released BenchProofer, an open-source tool that tries to fix a quiet weakness in how the industry scores coding agents. Today, a task is usually marked solved if the agent's patch passes a suite of tests. The problem, the BenchProofer team writes, is simple. An agent can pass every test and still be wrong on inputs no test happens to check. It can also learn to game the tests instead of solving the underlying problem.

That would be a measurement complaint if the numbers stayed on a leaderboard. The authors argue they no longer do. Companies use these scores to make product decisions, and they increasingly use them as the reward signal when they further train agents with reinforcement learning. If the score is wrong, the training can reward the wrong behavior.

[1]

BenchProofer starts from a task that already has a known reference fix. From that fix it tries to build three artifacts. The first is a formal specification: a precise statement, written so a machine can check it, of what a correct solution must do. In the post's simple example, a function that finds the largest number in a list must return a value that is in the list, and no element may be larger than it. The second artifact is a reference implementation written in a language a verifier can handle, such as Lean. The third is a proof that this implementation meets the specification, under stated assumptions, which a verifier then checks. Tests and audits are used to check that the verified implementation still corresponds to the original repository code.

Amazon is also releasing SWE-Proof, which adds these artifacts to all 500 tasks in SWE-Bench Verified, a standard set of real bug-fix tasks drawn from GitHub repositories. The point of the release is to let other labs rerun the stricter check, rather than only reading Amazon's own scores.

[1]

The result that should change how a leaderboard is read is not a new state-of-the-art percentage. It is the error rate among patches that already passed the tests. In Amazon's study, the audit flagged nearly one in three of Claude Opus 4.8's test-passing patches because their behavior differed from the reference fix. That automated flag is not the same thing as a confirmed bug. In a sample that humans then reviewed, about half of the flagged patches, 21 of 41, had real bugs. The rest of the flags were not confirmed as bugs in that sample, which is a limit on the audit, not a reason to ignore the confirmed ones.

The team also reports that agents do better on the stricter measure when they are given the specification in addition to the tests. Claude Opus 4.8's verified pass rate rose from 84.6% to 93.6%. GPT-5.5 rose from 79.6% to 94.4%. That is useful, and it should be read narrowly. It shows that a clearer, machine-checkable description of the task helps these agents. It does not show that ordinary SWE-Bench Verified scores, without that specification, were measuring the same thing.

[1]

A fair objection is that formal methods have promised to replace tests before, and have often remained too expensive for messy, real repositories. BenchProofer does not claim to work from a bug report alone. It needs a reference solution to build the specification, so it is a way to harden existing benchmarks, not a general substitute for tests on new tasks. Amazon ran the evaluations. The human review covered 41 flagged patches, not the entire flagged set. Some flags will be mismatches in specification, not user-visible bugs. None of that is hidden in the post, and the public SWE-Proof release is the right way to let others argue with the result.

[1]

Those limits do not rescue the habit of treating a test-passing patch as a clean unit of progress. If nearly a third of one strong model's test-passing patches were flagged, and half of a reviewed sample were real bugs, then a one-point move on SWE-Bench Verified is a weaker signal than the leaderboard typography suggests. That weakness is more serious if the same scores are being used as a training reward. An agent that learns to pass tests can be rewarded for a patch that is still wrong on untested inputs.

The thing to watch over the next six months is whether SWE-Proof is adopted as a second number next to SWE-Bench Verified, and whether other labs reproduce the "nearly one in three" flag rate, or a comparable human-confirmed bug rate, on the same 500 tasks. If those replications do not appear, Amazon will have shipped a useful tool and a vendor-run warning. If they do appear, the default way of scoring coding agents will need a qualifier that the tests alone do not settle the fix.

[1]