[Bug]: `/speckit.converge` can declare convergence without auditable acceptance-scenario coverage

[cadugevaerd]

Bug Description

/speckit.converge already requires one stable intent-inventory key per FR-###, buildable SC-###, and user-story acceptance scenario such as US1/AC2. However, its output contract exposes only a single undenominated line, Requirements / acceptance criteria checked.

An agent can therefore assess user stories in bulk, report a plausible aggregate (for example, 16 FR + 12 SC + 4 user stories), and still emit converged without exposing that only 4 user-story rows were checked instead of the 16 acceptance scenarios required by the inventory rule.

This is an observability/closure-contract defect rather than a missing inventory instruction: the prompt tells the agent what to assess, but the report cannot demonstrate whether it did so.

Steps to Reproduce

1. Run /speckit.converge after an implementation pass on a feature whose spec.md has multiple user-story acceptance scenarios.
2. Have the assessment aggregate user stories instead of evaluating each scenario key.
3. Observe that the report can satisfy the current summary-metrics text while omitting the number of scenarios assessed.
4. In the observed case, a second pass changed only the granularity from user stories to individual acceptance scenarios and surfaced a missing HIGH finding at US3/AC4; a third pass at the same granularity found no remaining gaps.

Expected Behavior

The convergence report should make completion independently auditable:

• report checked/total separately for buildable FRs, buildable SCs, user-story acceptance scenarios, plan decisions, and applicable Constitution MUST principles;
• derive each denominator from the artifacts loaded during the run, not from model memory;
• expose the stable inventory keys with a compact checked/gap/unassessed status;
• allow converged only when every applicable category is fully assessed and no findings remain;
• fail closed with a distinct incomplete_assessment result when any inventory key is unassessed, without appending a partial Convergence phase to tasks.md.

Actual Behavior

The current template requires the fine-grained inventory internally but only asks for aggregate summary labels. As a result, a report can look complete while concealing a mismatch such as 4 assessed user stories versus 16 required acceptance scenarios. A false converged would end the implementation/convergence loop with an unassessed HIGH requirement.

Specify CLI Version

0.14.3.dev0 (main at time of report)

AI Agent

Not applicable — this is a core command-template contract affecting all agent integrations.

Operating System

Linux 5.15.147-sun60iw2

Python Version

Python 3.13.5

Error Logs

No process error: the defect is a false-completeness reporting path.

Additional Context

Proposed focused change:

1. Update templates/commands/converge.md with denominated per-category coverage, an auditable key ledger, and the fail-closed outcome precedence incomplete_assessment → tasks_appended → converged.
2. Document the third outcome in the Agentic SDD reference.
3. Add a deterministic template-contract regression test plus manual command validation.

Searches of existing upstream Issues/PRs found no duplicate for converge coverage denominators or incomplete assessment.

AI disclosure: This report was prepared by Hermes Agent (OpenAI gpt-5.6-terra) on behalf of @cadugevaerd, using Carlos's observed three-pass evidence and current upstream source inspection.

[cadugevaerd]

Implementação proposta em #3754: adiciona métricas checked/total derivadas dos artefatos, ledger de chaves estáveis e o desfecho fail-closed incomplete_assessment antes de tasks_appended/converged.

[github-actions]

Bug assessment — converge-coverage-denominators: Valid · severity high

---

Bug Assessment: /speckit.converge can declare convergence without auditable acceptance-scenario coverage

• Slug: converge-coverage-denominators
• Created: 2026-07-29T00:26:39Z
• Source: issue [Bug]: /speckit.converge can declare convergence without auditable acceptance-scenario coverage #3752
• Verdict: valid
• Severity: high

Report (summarized)

Reported by @cadugevaerd (via Hermes Agent / gpt-5.6-terra). The /speckit.converge template requires the agent to build an internal inventory with one stable key per FR-###, SC-###, and US#/AC# acceptance scenario, but its Step 6 "Summary metrics" output block exposes only a single undenominated line — "Requirements / acceptance criteria checked" — with no checked/total per category. This allows an agent to aggregate across categories (e.g. reporting "16 FR + 12 SC + 4 user stories") and still emit converged even though only 4 user-story rows were checked instead of the 16 individual acceptance scenarios required by the inventory rule. The author observed a three-pass convergence cycle where a second pass at finer granularity surfaced a missing HIGH finding at US3/AC4 that the first pass had hidden.

No linked gist, log, or external URL to fetch.

Symptom

An agent running /speckit.converge can produce a summary that appears complete but hides a mismatch between the number of acceptance scenarios actually evaluated and the number required by the intent inventory built in Step 3. The current template has no fail-closed outcome for partial assessment, so a false converged can end the implementation loop while HIGH requirements remain unassessed.

Reproduction

1. Create a feature whose spec.md contains multiple user stories, each with multiple acceptance scenarios (e.g. 3 user stories × ~5 AC each = ~15 US#/AC# keys).
2. Run /speckit.converge after an implementation pass.
3. Observe the Step 6 summary: the agent can report "Requirements / acceptance criteria checked: 3 user stories (16 FRs, 4 SCs)" without enumerating per-AC coverage.
4. The template allows converged to be emitted because no denominator constraint is enforced in the output contract.
5. [NEEDS CLARIFICATION: exact model/agent combination and spec.md structure that triggered the three-pass observation — provided as narrative but no artifact attached]

Suspected Code Paths

• templates/commands/converge.md:188–193 — The "Summary metrics" bullet list in Step 6 is the direct site of the defect. The line "Requirements / acceptance criteria checked" is undenominated and does not require checked/total breakdowns per category (FR, SC, US/AC).
• templates/commands/converge.md:133–143 — Step 3 "Build the Intent Inventory" correctly requires per-key inventory (US1/AC2-style keys), but there is no link back to the Step 6 output contract that enforces those keys are surfaced/counted in the report.
• templates/commands/converge.md:197–225 — Step 7 defines only two outcomes: tasks_appended and converged. There is no incomplete_assessment outcome to fail closed when the agent has not assessed all inventory keys.

Root Cause Hypothesis

The template separates assessment logic (Step 3–5, which correctly mandates per-key inventory) from output contract (Step 6, which only requests narrative aggregates). Because LLM agents interpret the Step 6 output schema as the authoritative report format, they can satisfy it by collapsing per-key results into category-level prose without violating any explicit instruction. The absence of a third outcome (incomplete_assessment) means the only gate for converged is zero actionable findings — not full inventory coverage. Confidence: high — the template text directly confirms the mismatch between Steps 3 and 6.

Proposed Remediation

Preferred: Update Step 6 of templates/commands/converge.md to replace the single "Requirements / acceptance criteria checked" line with per-category checked/total metrics derived explicitly from the inventory constructed in Step 3. Add a compact key-ledger table (one row per inventory key with status: ✅ satisfied, ⚠️ gap → Finding Fxx, or — unassessed) so the report is independently auditable. Add a third outcome incomplete_assessment (between the existing tasks_appended and converged branches) in Step 7: if any inventory key carries unassessed status, the command MUST halt with incomplete_assessment and must NOT append a Convergence phase to tasks.md. The author's proposed change in Additional Context aligns with this approach.

Alternatives:

• Lighter approach: Require only the checked/total denominator per category in Step 6 without a full key ledger. This reduces verbosity but leaves gaps harder to audit (no key-level trace). Trade-off: lower agent overhead, but a sufficiently confused model could still under-count.
• Fail-open with warning: Emit a partial_assessment outcome that appends tasks but flags the incomplete coverage. Trade-off: avoids hard stops but does not prevent converged from being declared after a follow-up pass that also under-counts.

Files likely to change:

• templates/commands/converge.md — Step 6 summary metrics block and Step 7 outcome branches

Tests to add or update:

• A deterministic regression test that parses the rendered converge.md template and asserts the presence of denominator syntax (e.g. checked/total) in the Step 6 output schema and the incomplete_assessment outcome in Step 7.
• A manual command validation checklist (as noted in the issue's Additional Context).

Risks & Considerations

• Template contract change is additive for compliant agents: agents that already enumerate per-key status will continue to work; agents that aggregate will now produce an incomplete_assessment result and surface gaps they previously hid — this is the intended behavior.
• Verbosity increase: requiring a key ledger in every converge report will produce longer in-session output for large specs. Consider making the ledger compact (single-char status per key) or only emitting it when there are gaps.
• Backward compatibility: existing projects that have tasks.md files with prior Convergence phases are unaffected — the change is to the prompt template, not to tasks.md format.
• No API breakage, no migrations, no security risk — this is a pure command-template change.

Open Questions

• [NEEDS CLARIFICATION: What is the minimum spec.md structure (number of user stories and AC) needed to reliably reproduce the aggregation collapse behavior?]
• [NEEDS CLARIFICATION: Should the key ledger be required even when converged (for auditability), or only when gaps exist?]
• [NEEDS CLARIFICATION: Issue mentions PR fix: make converge coverage auditable #3754 as the proposed implementation — is that PR already opened and ready for review?]

Generated by 🐛 Assess Bug from Labeled Issue for issue #3752 · 79.6 AIC · ⌖ 23 AIC · ⊞ 35.7K · ◷