Skip to content

AFDocs fixes: refresh known-exceptions reference after 2026-07-31 audit - #435

Draft
oz-by-warp[bot] wants to merge 1 commit into
mainfrom
afdocs-audit-exceptions-refresh
Draft

AFDocs fixes: refresh known-exceptions reference after 2026-07-31 audit#435
oz-by-warp[bot] wants to merge 1 commit into
mainfrom
afdocs-audit-exceptions-refresh

Conversation

@oz-by-warp

@oz-by-warp oz-by-warp Bot commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Ran the AFDocs scorecard against docs.warp.dev and found no genuine, fixable site issues. This PR refreshes the audit skill's exceptions reference, which had drifted from what the audit actually reports.

Audit result: 79/100 (C+) — 23 checks: 16 pass, 4 fail, 1 warn, 2 skip.

No site source changed (src/, astro.config.mjs, vercel.json, and content are all untouched), so there is nothing to rebuild or redeploy.

Why no site fixes

Every failure and the one warning traced back to an allowlisted platform limitation, a checker false positive, or page-sampling variance. I ran the audit twice and read the AFDocs check implementations to confirm each one:

Check Status Finding
llms-txt-links-markdown fail (both runs) 13 of 15 same-origin llms.txt links serve markdown (87%) against a 90% threshold. The only two that don't are openapi.yaml and openapi.json, and toMdUrls() returns no .md candidates for non-page extensions, so they can never satisfy the check.
content-negotiation fail (both runs) Vercel serves static files before rewrites. Both remediations the afdocs-fix skill suggests are already committed (src/middleware.ts and the Accept: text/markdown rewrites in vercel.json) and still never fire.
markdown-content-parity fail (both runs) Turndown escaping numbered headings (### 2\. …). All flagged diffs are numbered heading text; content is intact.
page-size-html fail, then pass Sampling variance across the three intentionally-long pages.
llms-txt-directive-html warn, then pass AFDocs pulls openapi.yaml / openapi.json into its page sample via llms.txt; neither is HTML.

For the directive warning I crawled all 352 sitemap pages using AFDocs' own detection logic and confirmed the llms.txt directive is present on every page and buried past the 50% threshold on none.

I deliberately did not clear llms-txt-links-markdown. The only ways to reach 90% are deleting an OpenAPI link or padding llms.txt with extra links — both degrade the file to satisfy a scorecard rather than improving the agent experience. Every link that represents documentation prose already serves markdown (13/13).

Changes

references/known-exceptions.md:

  • content-start-position and markdown-url-support now pass — documented why, and kept the entries so a re-appearance is recognized as known noise rather than a new bug.
  • content-negotiation — recorded that the suggested middleware and rewrites are already in place, so a future agent doesn't re-add them.
  • llms-txt-links-markdown — replaced the vague "YAML/JSON are classified as non-markdown" note with the actual mechanism and the 13/15 threshold math.
  • llms-txt-directive-html / -md — new entry covering the OpenAPI page-sample artifact, with the full-crawl verification.
  • page-size-* — refreshed measurements. The three pages over 100K markdown chars are open-source-licenses (~1.37M), changelog/2026 (~154K), and privacy (~112K). Dropped the stale all-settings warning; it is ~40K now.
  • markdown-content-parity — refreshed the affected-page list, kept the prior sample for history.
  • Added a Sampling variance section recording which checks fail deterministically.

SKILL.md:

  • Added a short Sampling variance subsection instructing a re-run before reporting a newly flagged per-page check as a regression. llms-txt-directive-html and page-size-html flipped between the two runs with no site changes, so a single run isn't enough to call a regression.

Testing

  • Ran node .agents/skills/afdocs-audit/scripts/afdocs_audit.mjs twice; preflight confirmed no Vercel Firewall challenge, so both audits are valid.
  • Verified every claim in the reference against live responses: llms.txt link content types, .md availability across the sitemap, directive presence on all 352 pages, and markdown character counts for the oversized pages.
  • No build needed — changes are confined to .agents/, which is outside the Astro content pipeline.

Conversation: https://app.warp.dev/conversation/2597d1b1-9c4f-466e-b6af-80ced35eb448

Run: https://oz.warp.dev/runs/019fb730-0148-7ef8-ae20-b0f4e7eb4a1c

This PR was generated with Oz.

Ran the AFDocs scorecard against docs.warp.dev twice (afdocs@0.18.7).
Score 79/100 (C+): 16 pass, 4 fail, 1 warn, 2 skip. Every failure traced
back to an allowlisted platform limitation, a checker false positive, or
page-sampling variance, so no site changes were needed.

The exceptions reference had drifted from reality, which risks future
audits either re-attempting fixes that are already committed or treating
allowlisted noise as a regression. Updated it to match what the audit
actually reports now:

- content-start-position, markdown-url-support: now pass. Documented why,
  and kept the entries so a re-appearance is recognized as known noise.
- content-negotiation: recorded that both remediations the afdocs-fix
  skill suggests (src/middleware.ts and the vercel.json Accept-header
  rewrites) are already committed and still never fire on Vercel's static
  hosting, so they are not re-added.
- llms-txt-links-markdown: replaced the vague "YAML/JSON are classified as
  non-markdown" note with the real mechanism. 13 of 15 same-origin llms.txt
  links serve markdown (87%) against a 90% threshold; the only two that
  don't are openapi.yaml and openapi.json, which toMdUrls() can never
  satisfy. Noted that neither deleting a link nor padding llms.txt is an
  acceptable fix.
- llms-txt-directive-html / -md: new entry. The intermittent "2 missing"
  warn comes from AFDocs pulling openapi.yaml and openapi.json into its
  page sample via llms.txt. Verified against all 352 sitemap pages, using
  AFDocs' own detection logic, that the directive is present on every page
  and buried on none.
- page-size-*: refreshed measurements. The three pages over 100K markdown
  chars are open-source-licenses (~1.37M), changelog/2026 (~154K), and
  privacy (~112K). Dropped the stale all-settings warning; it is ~40K now.
- markdown-content-parity: refreshed the affected-page list and kept the
  prior sample for history.
- Added a sampling-variance section to both the skill and the reference.
  llms-txt-directive-html and page-size-html flipped between the two runs
  with no site changes, so a single run is not enough to call a regression.

Co-Authored-By: Oz <oz-agent@warp.dev>
@cla-bot cla-bot Bot added the cla-signed label Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant