Skip to content

AFDocs fixes: refresh known-exceptions reference after 2026-07-31 audit #429

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rachaelrenk wants to merge 1 commit into from
+58 −39
Closed

AFDocs fixes: refresh known-exceptions reference after 2026-07-31 audit #429

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .agents/skills/afdocs-audit/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,10 @@ Before reporting, cross-reference every issue against the known exceptions in `r

Only include a section if its count is > 0. Never list allowlisted issues under "Remaining."

### Sampling variance

AFDocs samples ~50 of ~352 sitemap pages per run, so per-page checks can flip between `pass`, `warn`, and `fail` with no site changes. Re-run the audit before reporting a newly flagged per-page check as a regression, and report a check as reproducible only when it fails on both runs. See the "Sampling variance" section in `references/known-exceptions.md` for which checks fail deterministically.

## Invalid audits (Vercel Firewall challenge)

Before trusting any failures, confirm the audit was actually able to reach the
Expand Down
93 changes: 54 additions & 39 deletions .agents/skills/afdocs-audit/references/known-exceptions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,70 +2,85 @@

This file lists checks from the afdocs-audit skill that may flag as warnings or failures but are expected and intentional. When reporting audit results, classify these as "Allowlisted" rather than "Remaining."

**Last verified**: 2026-07-31 against `afdocs@0.18.7` (two consecutive runs on `https://docs.warp.dev`).

## Sampling variance

AFDocs samples ~50 of ~352 sitemap pages per run, so which pages surface — and whether a check reports `pass`, `warn`, or `fail` — varies between runs. Two consecutive runs on 2026-07-31 produced different results for `llms-txt-directive-html` (warn, then pass) and `page-size-html` (fail, then pass) with no site changes in between.

Before treating a newly flagged check as a regression, re-run the audit. Only `content-negotiation`, `llms-txt-links-markdown`, and `markdown-content-parity` failed on both runs.

## content-negotiation

**Expected status**: fail
**Reason**: Vercel serves static files before evaluating rewrites. Since all doc pages are pre-rendered as static HTML, header-conditional rewrites (`Accept: text/markdown`) never fire — the CDN matches the static file first and returns HTML. Astro's `middlewareMode: 'edge'` also doesn't work because the edge function's `next()` forwards to `/_render`, which doesn't exist for static pages.
**Already attempted**: Both remediations the `afdocs-fix` skill suggests are already committed and still don't fire — `src/middleware.ts` (Astro middleware using `shouldServeMarkdown()`) and the `accept: .*text/markdown.*` header rewrites in `vercel.json`. Verified 2026-07-31: 0 of 48 sampled pages returned markdown. Do not "fix" this again by re-adding middleware.
**Mitigation**: The practical agent experience is already good despite this check failing:
- The llms.txt directive on every page tells agents that `.md` URLs are available
- `<link rel="alternate" type="text/markdown">` in `<head>` provides a machine-readable signal
- Agents that read either of these (Claude Code, Cursor, OpenCode) get clean markdown
- The `.md` URL convention works for 92%+ of pages
- The `.md` URL convention works for every Starlight doc page (48/48 sampled)
**Action**: No fix available on Vercel's static hosting. Would require migrating to a platform where middleware runs before static file serving (e.g., Cloudflare Pages). Accept as a platform limitation.

## content-start-position
## llms-txt-links-markdown

**Expected status**: fail or warn
**Reason**: Sampled pages may have content starting past 50% of the HTML output. This is inherent to Starlight's layout — sidebar navigation, header markup, and JavaScript/CSS precede the `<main>` content area.
**Mitigation**: The llms.txt directive, `<link rel="alternate" type="text/markdown">` in `<head>`, and `Accept: text/markdown` content negotiation middleware all steer agents to the clean markdown version, bypassing the HTML boilerplate entirely.
**Action**: No fix needed. This is a structural property of Starlight sites.
**Expected status**: fail
**Reason**: False positive caused by a threshold edge, not by a content gap. The check requires ≥90% of same-origin llms.txt links to serve markdown. Warp's llms.txt has 15 same-origin links: 13 `.txt` documentation sets (which AFDocs content-sniffs and correctly classifies as markdown) plus `openapi.yaml` and `openapi.json`. That yields 13/15 = 87%, just under the threshold. AFDocs also reports "no markdown alternatives detected" because `toMdUrls()` returns no `.md` candidates for non-page file extensions — so the two OpenAPI links can never satisfy the check, no matter what the site does.
**Affected links** (2 of 15 in llms.txt, configured via `optionalLinks` in `astro.config.mjs`):
- `https://docs.warp.dev/openapi.yaml`
- `https://docs.warp.dev/openapi.json`
**Action**: No fix needed. The OpenAPI specs are intentionally YAML/JSON and don't need markdown variants. Do not delete either link or pad llms.txt with extra links to clear the 90% threshold — both would degrade the file to satisfy a scorecard. Every link that represents documentation prose already serves markdown (13/13).

## llms-txt-directive-html / llms-txt-directive-md

**Expected status**: pass, with intermittent `warn` reporting "2 missing" or "2 had no markdown version"
**Reason**: False positive from page discovery, not a missing directive. AFDocs builds its page sample from the sitemap *and* from llms.txt links, and the top-level llms.txt walk in `walkAggregateLinksWithOriginals()` does not filter non-page file extensions. That pulls `openapi.yaml` and `openapi.json` into the sample as if they were pages. Neither is HTML, so the directive check records them as "not found," and neither has a `.md` variant, so the markdown check records them as "no markdown version."
**Verification**: A full crawl of all 352 sitemap pages on 2026-07-31, using AFDocs' exact detection logic (`LINK_PATTERN` / `TEXT_PATTERN` against the body with `<nav>`, `<script>`, and `<style>` stripped), found the directive present on every page, and buried past the 50% threshold on none. The directive is injected for Starlight pages by `src/components/CustomHeader.astro` and for `/api` by `src/pages/api.astro`.
**Related signal**: `llms-txt-coverage` passes but notes "2 llms.txt links not in sitemap" — the same two OpenAPI URLs.
**Action**: No fix needed. If this warn appears, confirm the count is 2 and that the missing entries are the OpenAPI specs before investigating further.

## markdown-content-parity

**Expected status**: warn or fail (varies by sample; a single flagged page can tip the check to fail)
**Reason**: False positive. The "missing" segments are numbered heading text like "2. Tabbed File Viewer" where Turndown correctly escapes the period (`### 2\. Tabbed File Viewer`) to prevent markdown parsers from interpreting it as a list item. The content IS present in the markdown — the AFDocs checker's text comparison doesn't account for markdown escaping. The specific page(s) flagged vary run to run because AFDocs samples a subset of pages.
**Affected pages** (as of 2026-05-05):
- `/platform/triggers/scheduled-agents-quickstart/` — step headings
- `/platform/integrations/github-actions/` — numbered use case headings
- `/support-and-community/troubleshooting-and-support/troubleshooting-login-issues/` — URLs with special chars
- `/reference/cli/quickstart/` — optional step headings
- `/guides/getting-started/welcome-to-warp/` — numbered section headings
- `/terminal/editor/vim/` — "See Vim docs:" link text
- `/guides/getting-started/10-coding-features-you-should-know/` — numbered feature headings
**Action**: No fix needed. Content is intact.

## llms-txt-links-markdown

**Expected status**: fail
**Reason**: False positive. The llms.txt file links to `openapi.yaml` and `openapi.json` (the Oz Agent API OpenAPI spec) in its Optional section. The `afdocs` checker classifies YAML/JSON files as "non-markdown" (the same bucket as HTML pages) and looks for `.md` variants that don't exist and shouldn't. Machine-readable spec files have no markdown equivalent by design.
**Affected links** (2 of 14 in llms.txt):
- `https://docs.warp.dev/openapi.yaml`
- `https://docs.warp.dev/openapi.json`
**Action**: No fix needed. The OpenAPI specs are intentionally YAML/JSON and don't need markdown variants.
**Affected pages** (2026-07-31 run: 41 pass, 8 warn, 1 fail, avg 2% missing):
- `/guides/configuration/creating-rules-for-agents/` — numbered walkthrough headings (fail bucket, 24% "missing" across 4 segments)
- `/guides/agent-workflows/run-a-software-factory-in-the-cloud/` — numbered step headings
- `/agent-platform/local-agents/interacting-with-agents/terminal-and-agent-modes/` — numbered setup headings
- `/enterprise/getting-started/getting-started-developers/` — numbered onboarding headings
- `/agent-platform/capabilities/skills/` — file-listing bullets (`check_links.py`, `config.yaml`)
**Earlier samples** (2026-05-05) additionally flagged `/platform/triggers/scheduled-agents-quickstart/`, `/platform/integrations/github-actions/`, `/support-and-community/troubleshooting-and-support/troubleshooting-login-issues/`, `/reference/cli/quickstart/`, `/guides/getting-started/welcome-to-warp/`, `/terminal/editor/vim/`, and `/guides/getting-started/10-coding-features-you-should-know/`.
**Action**: No fix needed. Content is intact. Confirm any newly flagged page shows escaped-period or list-punctuation diffs in `sampleDiffs` before treating it as a real gap.

## page-size-markdown / page-size-html

**Expected status**: fail (both checks can fail as intentionally-long reference pages grow)
**Reason**: A small set of pages are intentionally long and cannot be meaningfully split. They now exceed the AFDocs 100K-char hard-fail threshold, so `page-size-markdown` and/or `page-size-html` fail on any run that samples one of them. The `afdocs-fix` skill explicitly treats page-size as a do-not-auto-fix editorial concern.
**Affected pages** (as of 2026-07-10):
- `/support-and-community/community/open-source-licenses/` — auto-generated third-party license text (~1.37M md chars). Regenerated by the `release_updates` skill; any manual split would be overwritten. Drives the `page-size-html` max (396K HTML → 121K markdown).
- `/changelog/2026/` — current-year changelog (~113–130K md chars). Already split by year; the current year grows until year-end rollover. Drives the `page-size-markdown` failure.
- `/support-and-community/privacy-and-security/privacy/` — comprehensive privacy/legal content (~113K md chars). Kept whole for legal completeness.
- `/terminal/settings/all-settings/` — comprehensive settings reference. Below the 100K threshold as of this date but historically borderline; monitor.
**Sampling note**: AFDocs samples ~49–50 of ~345 sitemap pages per run, so which of these large pages surfaces (and whether a check reports fail or warn) varies between runs. Counts in the report will fluctuate accordingly.
**Expected status**: `page-size-html` fail or pass depending on sampling; `page-size-markdown` currently pass
**Reason**: A small set of pages are intentionally long and cannot be meaningfully split. They exceed the AFDocs 100K-char hard-fail threshold, so `page-size-html` fails on any run that samples one of them. The `afdocs-fix` skill explicitly treats page-size as a do-not-auto-fix editorial concern.
**Affected pages** (markdown character counts measured 2026-07-31):
- `/support-and-community/community/open-source-licenses/` — auto-generated third-party license text (~1.37M md chars, ~7.8M HTML chars). Regenerated by the `release_updates` skill; any manual split would be overwritten. Excluded from `llms-small.txt` and `llms-full.txt` via `exclude` in `astro.config.mjs` because it overflows `hast-util-to-text`.
- `/changelog/2026/` — current-year changelog (~154K md chars). Already split by year; the current year grows until year-end rollover.
- `/support-and-community/privacy-and-security/privacy/` — comprehensive privacy and legal content (~112K md chars). Kept whole for legal completeness.
These are the only three pages on the site over 100K markdown chars. `/terminal/settings/all-settings/` was previously flagged as borderline but is now ~40K md chars and no longer a concern.
**Action**: No fix. These pages are intentionally long (auto-generated, current-year changelog, or legal). Do not split. Re-evaluate `/changelog/2026/` at year-end when it rolls over to `/changelog/2027/`.

## section-header-quality
## content-start-position

**Expected status**: skip
**Reason**: Only evaluated when tab panels contain section headers. Most sampled pages with tabs don't have headers inside the tab panels, so the check is skipped.
**Action**: None needed.
**Expected status**: pass (previously fail or warn)
**Reason**: Historically flagged because Starlight's sidebar navigation, header markup, and JavaScript and CSS precede the `<main>` content area. Current AFDocs versions measure the content offset within the searchable `<body>` after stripping `<nav>`, `<script>`, and `<style>`, so Warp's pages now report a median start position of 0% (2026-07-31: all 50 sampled pages start within the first 10%).
**Mitigation** (if it regresses): the llms.txt directive, `<link rel="alternate" type="text/markdown">` in `<head>`, and the `.md` URL convention all steer agents to the clean markdown version, bypassing the HTML boilerplate entirely.
**Action**: None needed. Treat a re-appearance as a structural property of Starlight sites rather than a content problem.

## markdown-url-support

**Expected status**: fail (for `/api`)
**Reason**: `/api` is a custom Astro page (`src/pages/api.astro`), not a Starlight content page. The docs-markdown integration only generates `.md` variants for Starlight doc pages under `src/content/docs/`, so `/api.md` does not exist.
**Action**: No fix needed. This page is intentionally outside the Starlight content pipeline.
**Expected status**: pass (previously fail for `/api`)
**Reason**: `/api` is a custom Astro page (`src/pages/api.astro`), not a Starlight content page. The docs-markdown integration only generates `.md` variants for Starlight doc pages under `src/content/docs/`, so `/api.md` returns 404. Current AFDocs versions exclude `/api` from the eligible page set, so the check reports 48/48 (100%).
**Action**: No fix needed. This page is intentionally outside the Starlight content pipeline. If `/api` is flagged again, allowlist it rather than generating a stub `/api.md`.

## section-header-quality

**Expected status**: skip
**Reason**: Only evaluated when tab panels contain section headers. Most sampled pages with tabs don't have headers inside the tab panels, so the check is skipped (2026-07-31: 6 pages with tabs, no headers inside panels).
**Action**: None needed.

## auth-alternative-access

Expand Down
Loading