Skip to content

Verify claims about things other than tests: migrations, deploys, docs, dependencies #9

Description

@amrit110

The claim taxonomy in src/claims.rs covers tests, type checks,
lint, build, commit, push, and file writes. Agents routinely assert plenty else, and each of
these has recorded evidence that could back or refute it.

Each item below is a self-contained contribution: one claim pattern, one verifier, two tests.

  • "I ran the migration" — evidence: alembic upgrade, prisma migrate, rails db:migrate,
    django manage.py migrate, and what they printed
  • "I installed the dependency" — evidence: a package-manager command and the manifest
    actually changing (package.json, pyproject.toml, Cargo.toml)
  • "I updated the docs" — evidence: a write to a docs path; flag "documented in the README"
    when the README was never touched
  • "I deployed" / "it's live" — evidence: a deploy command that succeeded
  • "I reverted that" — evidence: git revert/git checkout/git restore, or an edit
    restoring the earlier content
  • "I removed the debug logging" — evidence: no console.log/print/dbg! remains in
    the added lines. This one is checkable directly from the edit payloads.
  • "the endpoint returns 200" — evidence: a curl/httpie call and its status
  • "I opened a PR" — evidence: gh pr create and the URL it printed
  • "no secrets were committed" — evidence: added lines containing key-shaped strings

How to add one

  1. Add a variant to ClaimKind and a pattern in all_patterns().
  2. Extend the evidence ledger in src/evidence.rs if the
    evidence is not already collected.
  3. Add a match arm in src/verify.rs returning a verdict and the
    line of output it rests on
    .
  4. Test both directions: the claim with evidence, and the claim without.

The trap to avoid

The patterns must reject intentions and hypotheticals. "I'll run the migration next" and "once
the migration runs" are not claims, and matching them makes the tool noisy enough that people
turn it off. is_hedged() and is_negated() exist for exactly this — extend them alongside any
new pattern, and add a test proving the hedged form is ignored.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions