Claims Need Replayable Evidence

An agent said the first two bytes of a fixture were an ELF header. The bytes were 4d5a. Reverify returned REFUTED, included the observed bytes, and exited with code 2.

That tiny result matters more than a polished summary. A later session can replay the claim against the same artifact and reach the same verdict. A paragraph that says “we checked the binary” cannot do that. It leaves the next agent to guess the input, tool version, and exact predicate.

The useful design pattern is small: keep a claim outside the model’s prose, run a deterministic checker, and save a receipt with the result. Store only verified facts as facts. Keep refuted and inconclusive results visible as constraints, not as truth.

The verifier must record enough information for another process to replay the decision.

One Artifact, Two Claims

I inspected Reverify 0.11.0 at commit f32ea84ccd6bb2aff90eca711ccfdbf04e6b8923, using Python 3.12.6 on Windows. It has a plain-Python core; optional packages add Capstone, Unicorn, LIEF, Z3, and angr. I used the core only.

The probe makes a four-byte private fixture, 4d5a9000, then asks the real CLI to assess two incompatible claims. From the checked-out repository:

python ..\..\..\public\blog\claims-need-replayable-evidence\reverify-probe.py .

claims=2 verified=1 refuted=1 grounded=False exit=2

The non-zero exit is deliberate. One claim matched and one did not, so the report could not count as trustworthy. The fixture is a synthetic byte string, not a meaningful executable. It tests the claim-and-verdict path without making a statement about a real program.

The raw CLI run makes the returned evidence clearer:

python -m reverify.cli verify 4d5a9000 `
  --claim '{"kind":"bytes_at","offset":0,"expected":"4d5a"}' `
  --claim '{"kind":"bytes_at","offset":0,"expected":"7f45"}' --json

{
  "total_claims": 2,
  "verified": 1,
  "refuted": 1,
  "grounded": false,
  "receipt": {
    "binary_sha256": "9f2981a7cc4d40a2a409dc895de64253acd819d7c0011c8e80b86fe899464e31",
    "binary_size": 4,
    "reverify": "0.11.0",
    "engines": {
      "disassembly": "pure-python",
      "emulation": "pure-python",
      "binary_parsing": "pure-python"
    },
    "replay": "reverify verify <file> --claims-file <claims.json>"
  }
}

This structure gives a handoff three things a transcript does not: the exact target hash, the checker configuration, and the command shape for replay. It also keeps the failing assertion. That last part prevents a later agent from recovering the false ELF statement from a compressed note and treating it as established context.

The Code Draws a Useful Line

verifier.py defines five outcomes: VERIFIED, REFUTED, INCONCLUSIVE, OBSERVED, and INVALIDATED. The separation is more useful than a boolean gate.

Verdict	Durable interpretation	Safe next action
VERIFIED	The stated predicate held for the recorded artifact and engine set.	Carry it with the receipt.
REFUTED	The checker found contradictory evidence.	Feed the evidence back; do not promote the claim.
INCONCLUSIVE	The selected tool cannot decide.	Change the tool or reduce the claim.
OBSERVED	The tool returned a value without judging an assertion.	Turn the value into a separate claim if needed.
INVALIDATED	A dependency failed.	Rebuild downstream reasoning.

Verifier.verify_all() evaluates each claim, applies dependency invalidation, then calculates whether the collection has enough information to be called grounded. It refuses a cheap route to green: trivial, duplicate, echoed, and self-referential claims can contribute zero weight. That is a sensible defence against an agent proving only that a file begins with bytes the prompt already supplied.

The receipt code uses the input SHA-256, byte length, package version, Python/platform values, backend list, generated time, and replay instruction. This is a real implementation detail, not a promise made in documentation. It turns verification output into an artifact another task can inspect.

The Test Surface Has a Gap

The repository contains a useful amount of executable coverage. I ran two focused modules through the standard library test runner:

python -m unittest discover -s reverify/tests -p "test_verifier.py" -q
python -m unittest discover -s reverify/tests -p "test_rollover.py" -q

Ran 27 tests in 0.101s
OK

Ran 8 tests in 0.158s
OK

The verification tests cover verdict counting, dependencies, claim formats, receipts, and the anti-trivia score. The rollover set exercises a mock session handoff. Those checks support the narrow interface above. They do not prove that all optional binary-analysis engines agree, because I did not install them.

I also ran the repository’s broader discovery command. It exposed a failing rollover expectation on this checkout:

python -m unittest reverify.tests.test_claude_rollover.Guard.test_request_without_a_session_is_keyed_by_cwd -v

FAIL: test_request_without_a_session_is_keyed_by_cwd
AssertionError: 'this directory' not found in
'rollover requested for session 01a0965a-...; it happens when this turn ends (...)'

The test expects a no-session request to be described as keyed by the working directory. The observed output uses a session identifier from the host environment. That does not invalidate byte verification, but it does make the context-handoff layer less settled than the README makes it sound. The project also emits a ResourceWarning in test_rollover.py for an unclosed checkpoint handle on this machine.

Evidence Has a Scope

Reverify supports more than byte predicates: PE/ELF/Mach-O parsing, instruction checks, protocol fields, emulation, behavioral comparisons, and source-function equivalence. Each path should still name the engine. A pure-Python decoder and an optional Capstone or LIEF backend do not provide identical coverage. A verified predicate means the selected checker had evidence for that predicate. It does not certify an entire binary or prove an agent's narrative.

The project gets this mostly right in its own vocabulary. INCONCLUSIVE is not silently upgraded to a pass. OBSERVED is not treated as an assertion. Dependencies can fall when their premise is refuted. Those are modest rules, but agent memory needs exactly this kind of boring bookkeeping.

Use the pattern for high-cost statements: a migration prerequisite, a schema field, a benchmark number, a deployment target, or a generated patch. Put the claim, evidence, replay command, and environment boundary beside the task record. Let the model write the explanation around that receipt. Do not let the explanation become the receipt.