From 0d006767b674d90cfd9ca74ea771ff84aa10c9f0 Mon Sep 17 00:00:00 2001 From: rachaelrenk Date: Fri, 31 Jul 2026 08:11:09 +0000 Subject: [PATCH] AFDocs fixes: refresh known-exceptions reference after 2026-07-31 audit Ran the AFDocs scorecard against docs.warp.dev twice (afdocs@0.18.7). Score 79/100 (C+): 16 pass, 4 fail, 1 warn, 2 skip. Every failure traced back to an allowlisted platform limitation, a checker false positive, or page-sampling variance, so no site changes were needed. The exceptions reference had drifted from reality, which risks future audits either re-attempting fixes that are already committed or treating allowlisted noise as a regression. Updated it to match what the audit actually reports now: - content-start-position, markdown-url-support: now pass. Documented why, and kept the entries so a re-appearance is recognized as known noise. - content-negotiation: recorded that both remediations the afdocs-fix skill suggests (src/middleware.ts and the vercel.json Accept-header rewrites) are already committed and still never fire on Vercel's static hosting, so they are not re-added. - llms-txt-links-markdown: replaced the vague "YAML/JSON are classified as non-markdown" note with the real mechanism. 13 of 15 same-origin llms.txt links serve markdown (87%) against a 90% threshold; the only two that don't are openapi.yaml and openapi.json, which toMdUrls() can never satisfy. Noted that neither deleting a link nor padding llms.txt is an acceptable fix. - llms-txt-directive-html / -md: new entry. The intermittent "2 missing" warn comes from AFDocs pulling openapi.yaml and openapi.json into its page sample via llms.txt. Verified against all 352 sitemap pages, using AFDocs' own detection logic, that the directive is present on every page and buried on none. - page-size-*: refreshed measurements. The three pages over 100K markdown chars are open-source-licenses (~1.37M), changelog/2026 (~154K), and privacy (~112K). Dropped the stale all-settings warning; it is ~40K now. - markdown-content-parity: refreshed the affected-page list and kept the prior sample for history. - Added a sampling-variance section to both the skill and the reference. llms-txt-directive-html and page-size-html flipped between the two runs with no site changes, so a single run is not enough to call a regression. Co-Authored-By: Oz --- .agents/skills/afdocs-audit/SKILL.md | 4 + .../references/known-exceptions.md | 93 +++++++++++-------- 2 files changed, 58 insertions(+), 39 deletions(-) diff --git a/.agents/skills/afdocs-audit/SKILL.md b/.agents/skills/afdocs-audit/SKILL.md index 0f3baf01b..8e4e35cde 100644 --- a/.agents/skills/afdocs-audit/SKILL.md +++ b/.agents/skills/afdocs-audit/SKILL.md @@ -53,6 +53,10 @@ Before reporting, cross-reference every issue against the known exceptions in `r Only include a section if its count is > 0. Never list allowlisted issues under "Remaining." +### Sampling variance + +AFDocs samples ~50 of ~352 sitemap pages per run, so per-page checks can flip between `pass`, `warn`, and `fail` with no site changes. Re-run the audit before reporting a newly flagged per-page check as a regression, and report a check as reproducible only when it fails on both runs. See the "Sampling variance" section in `references/known-exceptions.md` for which checks fail deterministically. + ## Invalid audits (Vercel Firewall challenge) Before trusting any failures, confirm the audit was actually able to reach the diff --git a/.agents/skills/afdocs-audit/references/known-exceptions.md b/.agents/skills/afdocs-audit/references/known-exceptions.md index a233305e0..0a959de53 100644 --- a/.agents/skills/afdocs-audit/references/known-exceptions.md +++ b/.agents/skills/afdocs-audit/references/known-exceptions.md @@ -2,70 +2,85 @@ This file lists checks from the afdocs-audit skill that may flag as warnings or failures but are expected and intentional. When reporting audit results, classify these as "Allowlisted" rather than "Remaining." +**Last verified**: 2026-07-31 against `afdocs@0.18.7` (two consecutive runs on `https://docs.warp.dev`). + +## Sampling variance + +AFDocs samples ~50 of ~352 sitemap pages per run, so which pages surface — and whether a check reports `pass`, `warn`, or `fail` — varies between runs. Two consecutive runs on 2026-07-31 produced different results for `llms-txt-directive-html` (warn, then pass) and `page-size-html` (fail, then pass) with no site changes in between. + +Before treating a newly flagged check as a regression, re-run the audit. Only `content-negotiation`, `llms-txt-links-markdown`, and `markdown-content-parity` failed on both runs. + ## content-negotiation **Expected status**: fail **Reason**: Vercel serves static files before evaluating rewrites. Since all doc pages are pre-rendered as static HTML, header-conditional rewrites (`Accept: text/markdown`) never fire — the CDN matches the static file first and returns HTML. Astro's `middlewareMode: 'edge'` also doesn't work because the edge function's `next()` forwards to `/_render`, which doesn't exist for static pages. +**Already attempted**: Both remediations the `afdocs-fix` skill suggests are already committed and still don't fire — `src/middleware.ts` (Astro middleware using `shouldServeMarkdown()`) and the `accept: .*text/markdown.*` header rewrites in `vercel.json`. Verified 2026-07-31: 0 of 48 sampled pages returned markdown. Do not "fix" this again by re-adding middleware. **Mitigation**: The practical agent experience is already good despite this check failing: - The llms.txt directive on every page tells agents that `.md` URLs are available - `` in `` provides a machine-readable signal - Agents that read either of these (Claude Code, Cursor, OpenCode) get clean markdown -- The `.md` URL convention works for 92%+ of pages +- The `.md` URL convention works for every Starlight doc page (48/48 sampled) **Action**: No fix available on Vercel's static hosting. Would require migrating to a platform where middleware runs before static file serving (e.g., Cloudflare Pages). Accept as a platform limitation. -## content-start-position +## llms-txt-links-markdown -**Expected status**: fail or warn -**Reason**: Sampled pages may have content starting past 50% of the HTML output. This is inherent to Starlight's layout — sidebar navigation, header markup, and JavaScript/CSS precede the `
` content area. -**Mitigation**: The llms.txt directive, `` in ``, and `Accept: text/markdown` content negotiation middleware all steer agents to the clean markdown version, bypassing the HTML boilerplate entirely. -**Action**: No fix needed. This is a structural property of Starlight sites. +**Expected status**: fail +**Reason**: False positive caused by a threshold edge, not by a content gap. The check requires ≥90% of same-origin llms.txt links to serve markdown. Warp's llms.txt has 15 same-origin links: 13 `.txt` documentation sets (which AFDocs content-sniffs and correctly classifies as markdown) plus `openapi.yaml` and `openapi.json`. That yields 13/15 = 87%, just under the threshold. AFDocs also reports "no markdown alternatives detected" because `toMdUrls()` returns no `.md` candidates for non-page file extensions — so the two OpenAPI links can never satisfy the check, no matter what the site does. +**Affected links** (2 of 15 in llms.txt, configured via `optionalLinks` in `astro.config.mjs`): +- `https://docs.warp.dev/openapi.yaml` +- `https://docs.warp.dev/openapi.json` +**Action**: No fix needed. The OpenAPI specs are intentionally YAML/JSON and don't need markdown variants. Do not delete either link or pad llms.txt with extra links to clear the 90% threshold — both would degrade the file to satisfy a scorecard. Every link that represents documentation prose already serves markdown (13/13). + +## llms-txt-directive-html / llms-txt-directive-md + +**Expected status**: pass, with intermittent `warn` reporting "2 missing" or "2 had no markdown version" +**Reason**: False positive from page discovery, not a missing directive. AFDocs builds its page sample from the sitemap *and* from llms.txt links, and the top-level llms.txt walk in `walkAggregateLinksWithOriginals()` does not filter non-page file extensions. That pulls `openapi.yaml` and `openapi.json` into the sample as if they were pages. Neither is HTML, so the directive check records them as "not found," and neither has a `.md` variant, so the markdown check records them as "no markdown version." +**Verification**: A full crawl of all 352 sitemap pages on 2026-07-31, using AFDocs' exact detection logic (`LINK_PATTERN` / `TEXT_PATTERN` against the body with `