tools/differential/compare.py classifies a diff by the first ledger rule that matches, where fields matches by subset (compare.py:469) and name_regex rules sort ahead of fields-only ones (compare.py:114). Nothing checks that the rule which claims a diff actually describes it, so a broad rule silently explains changes it says nothing about — and the run exits 0.
Found while reviewing #370, which hit it twice: once as a discovery, once by reproducing it.
Confirmed instances
1. fix(suffix-routing) claims the whole corpus. expected_since_1.4.0.toml:166 — the only fields-only rule in any ledger, fields = ["given","family","suffix"], no regex, so it matches all 751 names. Tallying classifier-of-record over every (name × field) pair, it owns 1639 of 5257 — 31% of the space. #370's diff landed on it before a rule was written:
classify('Mr. Van Nguyen', {'given','family'})
without a #367 rule -> fix(suffix-routing) "two-token name with unambiguous trailing suffix stays suffix"
2. fix(comma-family) matches on a bare comma. expected_since_1.4.0.toml:118 — name_regex = ",", fields = ["given","title","suffix"], reaching 236 corpus names and 715 (name × field) pairs. It latently absorbs #367's own comma-bearing class:
'Dr. Do Van Johnson, MD' moved ['given','title'] -> classified fix(comma-family)
No such name is in the corpus today, so this is latent — it becomes a green-on-regression the day one is added. That rule's prose already concedes it absorbed five CJK honorific rows it "has nothing to do with", which is documented; the bare-comma reach is not.
3. A blanket CJK rule shadows five to seven specific ones. expected_since_1.4.0.toml:62 and the 2.0.0 twin — name_regex is a bare CJK/Hangul/Kana character class, fields = ["given","middle","family"], matching all 97 CJK corpus names. It mechanically shadows fix(cjk-maiden-marker), fix(cjk-comma-compound), fix(cjk-honorific-suffix), fix(cjk-delimited-nickname), fix(cjk-fullwidth-paren-nickname), and in the 2.0.0 ledger also fix(#308/#312/#319/#320) and fix(#307/#308/#320). A future regression in Japanese or Korean given/family splitting is labelled with 2.1-era issue numbers and exits 0.
The structural gap: a rule that stops explaining anything is invisible
_CORPUS_CLAIMS in tests/v2/test_ledger_guards.py records what a rule's regex matches, which is independent of the parser. Reverting #370's fix and re-running showed 5 unit tests failing (good) but no ledger or differential failure — the 2.1.0 baseline reports 0 diffs, 0 unexplained, exit 0, with its only rule now explaining nothing. Deleting a rule is caught (test_every_rule_claims_the_recorded_share_of_the_corpus); a rule going inert is not.
Two cheap mechanical checks
Full over-claim detection is human judgement — no code decides whether "lone post-comma piece routes to suffix/title" describes Dr. Do Van Johnson, MD. But two checks are not judgement, and either would have surfaced instances 1 and 2 at authoring time:
- Report ambiguity.
classify() returns on first match. Have it collect every matching rule and print AMBIGUOUS 'Mr. Van Nguyen' -> [fix(#367), fix(suffix-routing)] alongside the classification. Exit code unchanged; the operator sees the overlap without having to delete a rule to discover it.
- A specificity floor for fields-only rules.
validate_rules (compare.py:362) already rejects a fields list naming all seven roles, but permits a fields-only rule with six. A fields-only rule reaching every corpus name is the shape most worth a warning, or an explicit declaration of unbounded reach.
A --strict mode asserting every rule fired at least once would close the third gap.
Note
#370 tightened its own fix(#367) rule after this was found — it originally reached 11 corpus names including Vincent van Gogh, which AGENTS.md:148 names as a standard regression canary — so the new rule is not an instance. The three above are pre-existing.
tools/differential/compare.pyclassifies a diff by the first ledger rule that matches, wherefieldsmatches by subset (compare.py:469) andname_regexrules sort ahead of fields-only ones (compare.py:114). Nothing checks that the rule which claims a diff actually describes it, so a broad rule silently explains changes it says nothing about — and the run exits 0.Found while reviewing #370, which hit it twice: once as a discovery, once by reproducing it.
Confirmed instances
1.
fix(suffix-routing)claims the whole corpus.expected_since_1.4.0.toml:166— the only fields-only rule in any ledger,fields = ["given","family","suffix"], no regex, so it matches all 751 names. Tallying classifier-of-record over every (name × field) pair, it owns 1639 of 5257 — 31% of the space. #370's diff landed on it before a rule was written:2.
fix(comma-family)matches on a bare comma.expected_since_1.4.0.toml:118—name_regex = ",",fields = ["given","title","suffix"], reaching 236 corpus names and 715 (name × field) pairs. It latently absorbs #367's own comma-bearing class:No such name is in the corpus today, so this is latent — it becomes a green-on-regression the day one is added. That rule's prose already concedes it absorbed five CJK honorific rows it "has nothing to do with", which is documented; the bare-comma reach is not.
3. A blanket CJK rule shadows five to seven specific ones.
expected_since_1.4.0.toml:62and the 2.0.0 twin —name_regexis a bare CJK/Hangul/Kana character class,fields = ["given","middle","family"], matching all 97 CJK corpus names. It mechanically shadowsfix(cjk-maiden-marker),fix(cjk-comma-compound),fix(cjk-honorific-suffix),fix(cjk-delimited-nickname),fix(cjk-fullwidth-paren-nickname), and in the 2.0.0 ledger alsofix(#308/#312/#319/#320)andfix(#307/#308/#320). A future regression in Japanese or Korean given/family splitting is labelled with 2.1-era issue numbers and exits 0.The structural gap: a rule that stops explaining anything is invisible
_CORPUS_CLAIMSintests/v2/test_ledger_guards.pyrecords what a rule's regex matches, which is independent of the parser. Reverting #370's fix and re-running showed 5 unit tests failing (good) but no ledger or differential failure — the 2.1.0 baseline reports 0 diffs, 0 unexplained, exit 0, with its only rule now explaining nothing. Deleting a rule is caught (test_every_rule_claims_the_recorded_share_of_the_corpus); a rule going inert is not.Two cheap mechanical checks
Full over-claim detection is human judgement — no code decides whether "lone post-comma piece routes to suffix/title" describes
Dr. Do Van Johnson, MD. But two checks are not judgement, and either would have surfaced instances 1 and 2 at authoring time:classify()returns on first match. Have it collect every matching rule and printAMBIGUOUS 'Mr. Van Nguyen' -> [fix(#367), fix(suffix-routing)]alongside the classification. Exit code unchanged; the operator sees the overlap without having to delete a rule to discover it.validate_rules(compare.py:362) already rejects afieldslist naming all seven roles, but permits a fields-only rule with six. A fields-only rule reaching every corpus name is the shape most worth a warning, or an explicit declaration of unbounded reach.A
--strictmode asserting every rule fired at least once would close the third gap.Note
#370 tightened its own
fix(#367)rule after this was found — it originally reached 11 corpus names includingVincent van Gogh, whichAGENTS.md:148names as a standard regression canary — so the new rule is not an instance. The three above are pre-existing.