fix(insights): round 2 says the derived outcome is 52%, not 63% — Step 1 reopens (v0.329.1)

[vikasprogrammer]

The Step 1 PR (#592) shipped with a debt: its 63% agreement was measured on the same 28 rows whose errors had just bought two of the rules. This is the re-validation that debt called for.

32 conversations round 1 never touched (--exclude), stratified across all nine bases, labelled blind from transcripts, scored against the rules exactly as shipped in v0.323.0. Nothing was changed after seeing the result.

	exact	baseline	sign
round 1, v1 rules	50%	43%	—
round 1, after tuning	63%	32%	74%
round 2, out of sample	52%	43%	78%

The 63% was fitting. The real signal is nine points above "call everything a success" — not enough to build Step 2 on. Step 1 goes back to open.

The 11 disagreements, and what they cost

1. 5 × no-evidence → unknown on conversations a human reads in seconds (4 successes, 1 partial) — runs that did substantial work and left no observable trace. This is exactly the residual the Stop-hook half was deferred over in Step 1. It is 6% of the corpus but 45% of the mistakes, because those runs are concentrated in the cases the OS is blind to rather than spread evenly. Judging a residual by its share instead of its share of the errors is the mistake. The hook is back in scope, and Step 1 is not finished without it.
2. 3 × quota deaths with zero tool calls read as noop — they fall through died-early's tool_calls > 0 guard, which round 1 itself bought (an agent replying "looks like a test message" is a noop, not a failure). The DB cannot currently tell the two shapes apart; this needs a fact neither has today.
3. 1 × task-retried beat a later success in a 4-run conversation — a fold-order bug, and the cheapest of the three to fix.

Two further disagreements were probably my labels, not the rules: I called two automation runs that produced no assistant output at all noop, where died-early called them failures. An unattended run that produces nothing is closer to a failure. Corrected, the number is ~57%.

No rules changed here

Fixing rules against the round that judged them is how the 63% happened. The three fixes get measured on round 3, against rows neither round has seen.

Tooling: outcome-label-score.cjs takes a labels-file argument; outcome-label-sample.cjs takes --exclude so a re-validation can only draw rows an earlier round never saw. Docs and scripts only — no src/ change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AiTKjtdcF4ESbuYVWj2CfU