diff --git a/.agents/skills/afdocs-audit/SKILL.md b/.agents/skills/afdocs-audit/SKILL.md index 0f3baf01..8e4e35cd 100644 --- a/.agents/skills/afdocs-audit/SKILL.md +++ b/.agents/skills/afdocs-audit/SKILL.md @@ -53,6 +53,10 @@ Before reporting, cross-reference every issue against the known exceptions in `r Only include a section if its count is > 0. Never list allowlisted issues under "Remaining." +### Sampling variance + +AFDocs samples ~50 of ~352 sitemap pages per run, so per-page checks can flip between `pass`, `warn`, and `fail` with no site changes. Re-run the audit before reporting a newly flagged per-page check as a regression, and report a check as reproducible only when it fails on both runs. See the "Sampling variance" section in `references/known-exceptions.md` for which checks fail deterministically. + ## Invalid audits (Vercel Firewall challenge) Before trusting any failures, confirm the audit was actually able to reach the diff --git a/.agents/skills/afdocs-audit/references/known-exceptions.md b/.agents/skills/afdocs-audit/references/known-exceptions.md index a233305e..0a959de5 100644 --- a/.agents/skills/afdocs-audit/references/known-exceptions.md +++ b/.agents/skills/afdocs-audit/references/known-exceptions.md @@ -2,70 +2,85 @@ This file lists checks from the afdocs-audit skill that may flag as warnings or failures but are expected and intentional. When reporting audit results, classify these as "Allowlisted" rather than "Remaining." +**Last verified**: 2026-07-31 against `afdocs@0.18.7` (two consecutive runs on `https://docs.warp.dev`). + +## Sampling variance + +AFDocs samples ~50 of ~352 sitemap pages per run, so which pages surface — and whether a check reports `pass`, `warn`, or `fail` — varies between runs. Two consecutive runs on 2026-07-31 produced different results for `llms-txt-directive-html` (warn, then pass) and `page-size-html` (fail, then pass) with no site changes in between. + +Before treating a newly flagged check as a regression, re-run the audit. Only `content-negotiation`, `llms-txt-links-markdown`, and `markdown-content-parity` failed on both runs. + ## content-negotiation **Expected status**: fail **Reason**: Vercel serves static files before evaluating rewrites. Since all doc pages are pre-rendered as static HTML, header-conditional rewrites (`Accept: text/markdown`) never fire — the CDN matches the static file first and returns HTML. Astro's `middlewareMode: 'edge'` also doesn't work because the edge function's `next()` forwards to `/_render`, which doesn't exist for static pages. +**Already attempted**: Both remediations the `afdocs-fix` skill suggests are already committed and still don't fire — `src/middleware.ts` (Astro middleware using `shouldServeMarkdown()`) and the `accept: .*text/markdown.*` header rewrites in `vercel.json`. Verified 2026-07-31: 0 of 48 sampled pages returned markdown. Do not "fix" this again by re-adding middleware. **Mitigation**: The practical agent experience is already good despite this check failing: - The llms.txt directive on every page tells agents that `.md` URLs are available - `` in `` provides a machine-readable signal - Agents that read either of these (Claude Code, Cursor, OpenCode) get clean markdown -- The `.md` URL convention works for 92%+ of pages +- The `.md` URL convention works for every Starlight doc page (48/48 sampled) **Action**: No fix available on Vercel's static hosting. Would require migrating to a platform where middleware runs before static file serving (e.g., Cloudflare Pages). Accept as a platform limitation. -## content-start-position +## llms-txt-links-markdown -**Expected status**: fail or warn -**Reason**: Sampled pages may have content starting past 50% of the HTML output. This is inherent to Starlight's layout — sidebar navigation, header markup, and JavaScript/CSS precede the `
` content area. -**Mitigation**: The llms.txt directive, `` in ``, and `Accept: text/markdown` content negotiation middleware all steer agents to the clean markdown version, bypassing the HTML boilerplate entirely. -**Action**: No fix needed. This is a structural property of Starlight sites. +**Expected status**: fail +**Reason**: False positive caused by a threshold edge, not by a content gap. The check requires ≥90% of same-origin llms.txt links to serve markdown. Warp's llms.txt has 15 same-origin links: 13 `.txt` documentation sets (which AFDocs content-sniffs and correctly classifies as markdown) plus `openapi.yaml` and `openapi.json`. That yields 13/15 = 87%, just under the threshold. AFDocs also reports "no markdown alternatives detected" because `toMdUrls()` returns no `.md` candidates for non-page file extensions — so the two OpenAPI links can never satisfy the check, no matter what the site does. +**Affected links** (2 of 15 in llms.txt, configured via `optionalLinks` in `astro.config.mjs`): +- `https://docs.warp.dev/openapi.yaml` +- `https://docs.warp.dev/openapi.json` +**Action**: No fix needed. The OpenAPI specs are intentionally YAML/JSON and don't need markdown variants. Do not delete either link or pad llms.txt with extra links to clear the 90% threshold — both would degrade the file to satisfy a scorecard. Every link that represents documentation prose already serves markdown (13/13). + +## llms-txt-directive-html / llms-txt-directive-md + +**Expected status**: pass, with intermittent `warn` reporting "2 missing" or "2 had no markdown version" +**Reason**: False positive from page discovery, not a missing directive. AFDocs builds its page sample from the sitemap *and* from llms.txt links, and the top-level llms.txt walk in `walkAggregateLinksWithOriginals()` does not filter non-page file extensions. That pulls `openapi.yaml` and `openapi.json` into the sample as if they were pages. Neither is HTML, so the directive check records them as "not found," and neither has a `.md` variant, so the markdown check records them as "no markdown version." +**Verification**: A full crawl of all 352 sitemap pages on 2026-07-31, using AFDocs' exact detection logic (`LINK_PATTERN` / `TEXT_PATTERN` against the body with `