fix(claim-evidence): refuse a check that cannot fail

[drewstone]

The hole

verdictFor graded a claim verified when its check exited zero and — only when an expect value was given — the output contained it. At rung 4 and above, where a claim asserts "a command reproduces a value", two evidence shapes passed that test without deciding anything. Both were found on a real corpus.

Shape 1 — a check with no expectation. check: true, no expect. The command exits zero, prints nothing, and the output test is skipped because there is nothing to look for. An agent minted a rung-4 claim exactly this way (probe2: rung4 with check only), and it was auto-promoted into a shared knowledge store as a novel finding.

Shape 2 — a check that prints its own expectation. check: echo '<the expectation>' with a matching expect. The command returns the string it is compared against, so the comparison can only succeed. Three pages in a shared store carry this shape.

The doctrine line these violate: an acceptance criterion that cannot fail is not one.

The change

At rung >= CHECKABLE_RUNG_THRESHOLD, both shapes now grade uncheckable — the check did not decide the claim. They are not contradicted, because nothing contradicted the claim. The lattice already had the right verdict; the function was not reaching it.

• A missing or empty expect yields uncheckable: "exit code alone cannot verify a claim at this rung — record the value the check must print".
• A constant-emitter check yields uncheckable with a note naming the shape.

gradeFor(evidence, execution) is added and returns { verdict, note }, so a grader can tell the author which shape was refused and what to record. verdictFor keeps its exact signature and returns the verdict alone, so no caller has to change.

What counts as a constant emitter

The definition is mechanical and deliberately narrow. A check is a constant emitter when:

• the whole command is true or :, or
• the whole command is one echo or printf whose arguments contain no command substitution ($( or a backtick), no pipe, no command separator, no redirection from a file, and no variable reference ($' and $" are quoting sigils, not variables).

Anything else is treated as a real check. echo "roots=$(grep -c root out.txt)" still verifies, and so does echo $ROOTS, because both read a value from outside the command line.

What this does NOT catch, and why

It cannot catch every check that cannot fail. A script that prints a hard-coded number, a command whose output an author copied into expect, and an echo behind a shell alias are all indistinguishable from a real check at this layer. Detecting those needs an independent party to re-derive the value, which is what rung 5 means — it is not a job static inspection of a command string can do.

So this refuses only the shapes that carry zero information on their face, where refusing costs the author nothing but a rewrite of the command. The limit is stated in the doc comment on isConstantEmitter so the next reader does not mistake it for a complete defence.

What is unchanged

• Below rung 4, nothing tightens. check: 'true' at rung 3 still verifies.
• An execution that decided the claim outranks both refusals: a missing input stays unrunnable, and a nonzero exit stays contradicted.
• Non-empty output that lacks the expectation stays contradicted; empty output with an expectation stays silent-check.

Known asymmetry, deliberately left

assertGradeableEvidence is the record-time gate and still accepts check: 'true' at rung 4; it only requires that some check exists. Grade time now refuses what record time admits. Closing that means changing a throw-behaviour contract at the recording boundary, which is a larger break than this fix, so it is named here rather than smuggled in.

Proof

Every new test was run against the unmodified source first. Failures reproduce both gamed shapes:

 × an exit code with no expectation decides nothing
   AssertionError: expected 'verified' to be 'uncheckable'
 × `true` and `:` as the whole check are constant emitters
   AssertionError: expected 'silent-check' to be 'uncheckable'
 × an echo that prints its own expectation is a constant emitter
   AssertionError: expected 'verified' to be 'uncheckable'
 × a printf that prints its own expectation is a constant emitter
   AssertionError: expected 'verified' to be 'uncheckable'
 × names the missing expectation            TypeError: gradeFor is not a function
 × names the constant-emitter shape         TypeError: gradeFor is not a function
 × carries no note when the check decided the claim

 Test Files  1 failed (1)
      Tests  7 failed | 12 passed (19)

The 12 that passed without the fix are the guards: command substitution still verifies, below-threshold behaviour is unchanged, and unrunnable/contradicted still outrank the new refusals. They prove the change is not a blanket tightening.

With the fix, the same file is green, and so is the repository:

pnpm lint       Checked 207 files in 183ms. No fixes applied.
pnpm typecheck  tsc --noEmit && tsc --noEmit -p tsconfig.contracts.json   (clean)
pnpm build      35 files, total: 2.65 MB — Build complete in 2227ms
pnpm test       Test Files  60 passed | 3 skipped (63)
                Tests  569 passed | 12 skipped (581)

[tangletools]

✅ Auto-approved drewstone PR — 77e72461

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

This approval is provisional. It rests on the audit running. If the audit cannot run — for example the CLI bridge rejects it — this approval is dismissed rather than left standing, so an unrun check never reads as a passing one.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-15T03:04:26Z}