diff --git a/.agents/skills/aeo_crosslink_audit/SKILL.md b/.agents/skills/aeo_crosslink_audit/SKILL.md index 5c7e8a3ba..c22a59c79 100644 --- a/.agents/skills/aeo_crosslink_audit/SKILL.md +++ b/.agents/skills/aeo_crosslink_audit/SKILL.md @@ -28,28 +28,39 @@ Do not: The following environment secrets should be set in the Oz cloud agent environment: +- `PEEC_PAT` — Peec Personal Access Token for MCP authentication. Create one at **app.peec.ai → Company → API Keys → Personal Access Tokens**. If unavailable or expired, the audit proceeds with GSC and docs-only signals and logs "Peec: unavailable" in the run output. - `SLACK_BOT_TOKEN` — Slack bot token for posting to `#growth-docs`. If unavailable, write the notification body to the run output instead and skip Slack posting. - `GROWTH_DOCS_SLACK_CHANNEL_ID` — Slack channel ID for `#growth-docs`. Find it in Slack by right-clicking the channel → Copy link (the ID begins with `C`). If unavailable, skip Slack posting. +The scheduled cloud agent must also include the Peec MCP server in its agent config (pass via `--mcp` flag or the agent config file `mcp_servers` key): + +```json +{ + "peec-ai": { + "url": "https://api.peec.ai/mcp", + "headers": { + "Authorization": "Bearer ${PEEC_PAT}" + } + } +} +``` + Do NOT print, log, commit, or include secret values in reports or Slack messages. ## Source data Use the smallest reliable set of source data needed to justify link changes: -- **Peec snapshot** - Check freshness before using any data: - 1. Read `generated_at` from `/workspace/buzz/aeo-snapshots/docs/agents-orchestration/latest.json`. - 2. If the file is missing, `generated_at` is absent, or the snapshot is **14 days old or older**: write the stale-snapshot report (see "Stale snapshot report" below), write a run log entry (step 7) with the appropriate `No-change reason` (see "Stale snapshot report" for exact wording), post the stale Slack alert (step 8), and exit. Do not continue the audit or open a PR. - 3. If the snapshot is fewer than 14 days old, read both `latest.json` and `latest.md` as source signals. These contain pre-exported Peec data (prompts, recommendations, source URLs, query vocabulary, and visibility scores) for agents, cloud agents, and orchestration. The snapshots are generated locally (where Peec OAuth works) and committed to the buzz repo so cloud agents can use them. +- **Peec** - Use the Peec MCP (configured in the agent with a Personal Access Token via the `PEEC_PAT` secret) to collect prompts, search queries, actions/recommendations, and source URLs for agents, cloud agents, and orchestration (last 30 days). Filter prompts and queries for relevance to the topic area. If the Peec MCP returns an error or is unavailable (missing `PEEC_PAT`, expired token, or connection failure), log "Peec: unavailable" in the run output and proceed with GSC and docs-only signals only. - **Google Search Console** - When available, use the environment's `GSC_SERVICE_ACCOUNT_CREDENTIALS_JSON` secret to inspect recent queries and pages related to agents, cloud agents, and orchestration. Never print, log, commit, or include the secret value in reports. If a GSC client requires a credentials file path, write the secret to a restricted temporary file, use it for the run, and remove it before finishing. - **Docs repo** - Search existing pages under `src/content/docs/` for relevant source pages, link targets, and related terminology. -If Google Search Console data is unavailable, say what could not be verified and proceed with Peec and docs-only analysis. If the Peec snapshot is stale or missing, exit via the staleness path above instead of proceeding. Do not invent source signals. +If Google Search Console data is unavailable, say what could not be verified and proceed with Peec and docs-only analysis. If Peec MCP is unavailable, log the unavailability and proceed with GSC and docs-only analysis. Do not invent source signals. -**Low-signal runs.** If neither live signal is usable for a run — the Peec snapshot has no usable prompt or recommendation data for these topics *and* Google Search Console is unavailable (missing credentials or a 403) — prefer a no-change report over a docs-only PR. Only open a PR in this case when there are at least 3 link additions that are each strongly grounded in existing on-page content and a clear reader journey. This prevents shipping weak, docs-only PRs that reviewers close. +**Low-signal runs.** If neither live signal is usable for a run — Peec MCP is unavailable *and* Google Search Console is unavailable (missing credentials or a 403) — prefer a no-change report over a docs-only PR. Only open a PR in this case when there are at least 3 link additions that are each strongly grounded in existing on-page content and a clear reader journey. This prevents shipping weak, docs-only PRs that reviewers close. ## Workflow -1. **Check snapshot freshness, then gather source signals.** Read `generated_at` from `latest.json`. If the snapshot is missing or 14 days old or older, write the stale-snapshot report, write a run log entry (step 7), post the stale Slack alert (step 8), and exit — do not proceed further. If fresh, read both snapshot files and use Google Search Console data, when available, to identify relevant user language, prompts, recommendations, or pages. +1. **Collect Peec signals and gather source signals.** Use the Peec MCP to collect prompts, search queries, actions/recommendations, and source URLs for agents, cloud agents, and orchestration (last 30 days). If the Peec MCP is unavailable, log "Peec: unavailable" and continue. Also collect Google Search Console data, when available, to identify relevant user language, prompts, recommendations, or pages. 2. **Search existing docs.** Look for pages under `src/content/docs/` that already mention or imply related concepts in agents, cloud agents, or orchestration. 3. **Identify link opportunities.** Find up to 5 internal cross-link opportunities where: - The source page already mentions or implies the related concept. @@ -214,38 +225,6 @@ Use this format: No-change reports stay in the Oz run output. The Oz run link is posted automatically to `#growth-docs` as part of step 8. -## Stale snapshot report - -If the Peec snapshot is missing or 14 days old or older, stop immediately. Write this report to the Oz run output: - -```text -## AEO cross-link audit — snapshot stale - -**Date:** YYYY-MM-DD -**Snapshot age:** [N days (generated YYYY-MM-DD) | file not found | generated_at field missing] -**Threshold:** 14 days - -The Peec snapshot is too old to support a high-confidence audit. No PR was opened and no docs were changed. - -**Action required:** -Run the `refresh-peec-aeo-snapshot` skill in a local Warp agent session where Peec MCP is authenticated. -Skill: buzz/.agents/skills/refresh-peec-aeo-snapshot/SKILL.md - -The audit will run normally on the next scheduled execution once a fresh snapshot is committed to the buzz repo. -``` - -Fill in the `Snapshot age` field as follows — do not invent values: -- File exists and `generated_at` is present: `N days (generated YYYY-MM-DD)` — compute `N` from today's date minus `generated_at`. -- File does not exist: `file not found`. -- File exists but `generated_at` is absent or unparseable: `generated_at field missing`. - -Use the same wording in the `No-change reason` field of the run log entry: -- File stale: `snapshot stale — N days old` -- File missing: `snapshot missing — file not found` -- Field missing: `snapshot missing — generated_at field absent` - -Then post the stale Slack alert (step 8). Exit. Do not write a no-change report. Do not open a PR. - ## Human review expectations The human reviewer should be able to understand the PR or no-change report without replaying the full run. Optimize the output for quick review: @@ -259,7 +238,7 @@ The human reviewer should be able to understand the PR or no-change report witho Prepend each new entry at the top of `.agents/logs/aeo_crosslink_audit_runs.md`, immediately after the `---` separator line. Use this format: ```markdown -## YYYY-MM-DD — [PR opened | No change | Snapshot stale] +## YYYY-MM-DD — [PR opened | No change] - **Run**: [Oz run URL if available, otherwise the run ID] - **Source signals**: Peec [available | unavailable], GSC [available | unavailable] @@ -267,7 +246,7 @@ Prepend each new entry at the top of `.agents/logs/aeo_crosslink_audit_runs.md`, - **Links proposed / added**: [N proposed, N added | N/A] - **Pages touched**: [comma-separated file paths | N/A] - **Themes**: [one sentence on recurring content gaps or topics observed, or "none observed"] -- **No-change reason**: [low confidence | lack of signals | snapshot stale — N days old | N/A] +- **No-change reason**: [low confidence | lack of signals | Peec unavailable | GSC unavailable | N/A] ``` Keep each entry to 7 fields and under 10 lines. Do not add narrative prose. @@ -295,17 +274,6 @@ No PR: [brief reason — e.g., "fewer than 2 high-confidence opportunities"] Oz run: [run URL] ``` -**Snapshot stale:** - -``` -⚠️ AEO crosslink audit · YYYY-MM-DD — Snapshot stale -Snapshot: [N days old (generated YYYY-MM-DD) | file not found | generated_at missing], threshold: 14 days -No audit ran. Refresh the snapshot before the next run. -How: run refresh-peec-aeo-snapshot in a local Warp session -Skill: buzz/.agents/skills/refresh-peec-aeo-snapshot/SKILL.md -Oz run: [run URL] -``` - Rules: - Post on every run, including no-change runs. - Never include raw secret values, personal access tokens, or credential file paths in the Slack message. diff --git a/.agents/skills/improve-aeo-crosslink-skill/SKILL.md b/.agents/skills/improve-aeo-crosslink-skill/SKILL.md index 4a5bdf01f..546f82d0a 100644 --- a/.agents/skills/improve-aeo-crosslink-skill/SKILL.md +++ b/.agents/skills/improve-aeo-crosslink-skill/SKILL.md @@ -17,8 +17,7 @@ Suggested cron: `0 17 1-7 * 1` (UTC) = first Monday of each month at 9am PT. ## Prerequisites -- Docs repo checked out at `main`, with at least 8 entries in `.agents/logs/aeo_crosslink_audit_runs.md` -- The standing log PR (`chore: aeo crosslink audit run log`) merged into `main` so the entries are present there. If it is unmerged, merge it first (or read the log from the `chore/aeo-crosslink-audit-log` branch) before analyzing. +- Docs repo checked out at `main` - `gh` CLI authenticated with write access to `warpdotdev/docs` - `SLACK_BOT_TOKEN` — for posting summary to `#growth-docs` - `GROWTH_DOCS_SLACK_CHANNEL_ID` — channel ID for `#growth-docs` @@ -31,6 +30,48 @@ Do not act if fewer than 8 entries exist. Write a "too early to analyze" notice ## Workflow +### 0. Merge the standing log PR + +Before reading the run log, ensure all accumulated entries are on `main` by merging the standing log PR. This is the PR from `chore/aeo-crosslink-audit-log` that the `aeo_crosslink_audit` agent continuously appends to. + +```bash +# Find the open log PR (there should be at most one) +OPEN_LOG_PR=$(gh pr list --repo warpdotdev/docs \ + --head chore/aeo-crosslink-audit-log \ + --state open \ + --json number \ + --jq '.[0].number' 2>/dev/null) + +if [[ -n "$OPEN_LOG_PR" ]]; then + # Safety check: only merge if the PR touches exactly the expected log file. + CHANGED_FILES=$(gh pr view "$OPEN_LOG_PR" --repo warpdotdev/docs --json files --jq '[.files[].path]') + ONLY_LOG=$(echo "$CHANGED_FILES" | python3 -c " +import json, sys +files = json.load(sys.stdin) +print('yes' if all(f == '.agents/logs/aeo_crosslink_audit_runs.md' for f in files) else 'no') +") + if [[ "$ONLY_LOG" == 'yes' ]]; then + gh pr merge "$OPEN_LOG_PR" --repo warpdotdev/docs --merge + # Non-destructive fast-forward: fails loudly if worktree is dirty or not fast-forwardable. + git fetch origin main + git merge --ff-only origin/main + else + echo "Log PR contains unexpected files — skipping merge, reading log from branch instead." + git fetch origin chore/aeo-crosslink-audit-log + git checkout origin/chore/aeo-crosslink-audit-log -- .agents/logs/aeo_crosslink_audit_runs.md + fi +fi +``` + +If the merge fails (conflict, permissions, or the branch is ahead of main in an unexpected way), log the failure to run output and read the log from the `chore/aeo-crosslink-audit-log` branch instead: + +```bash +git fetch origin chore/aeo-crosslink-audit-log +git checkout origin/chore/aeo-crosslink-audit-log -- .agents/logs/aeo_crosslink_audit_runs.md +``` + +Do not abort the skill run because the log PR could not be merged. Proceed with whatever log entries are available. + ### 1. Parse the run log Read all entries from `.agents/logs/aeo_crosslink_audit_runs.md`. For each entry, extract: @@ -51,9 +92,9 @@ Possible causes: - Scope (agents, cloud agents, orchestration) is too narrow and has been saturated - Peec or GSC data is consistently unavailable, reducing signal -**Peec snapshot consistently unavailable (5+ entries show "Peec: unavailable")** -Cause: snapshot files in `/workspace/buzz/aeo-snapshots/` are stale or the refresh cadence is too infrequent. -Fix: update the snapshot refresh instructions or cadence in `aeo_crosslink_audit/SKILL.md`. +**Peec consistently unavailable (5+ entries where `Source signals` shows `Peec unavailable`)** +Cause: the `PEEC_PAT` secret in the Oz environment is expired, revoked, or missing; or the Peec MCP config is incorrect. +Fix: verify the `PEEC_PAT` secret is valid at app.peec.ai → Company → API Keys → Personal Access Tokens. Renew or recreate it and update the Oz environment secret. Also confirm the scheduled agent config includes the Peec MCP server entry. **Links proposed but not added pattern (proposed > 0, added = 0 consistently)** Cause: the self-review step is rejecting candidates that have already passed the initial selection. Confidence rules may be miscalibrated.