Token usage investigation

[mnkiefer]

• Consider reports from the Token Audit and Optimizer workflows (agentic-ops bundle) and trends over time as shown in the plots here: [agentic-token-audit] Token Usage Audit — 2026-05-29 #35698
• Investigate token usage from 2026-05-01 to current date more closely and and attribute the trend to changes in the repository/workflows/etc.

[mnkiefer]

/scout history=full

[github-actions]

🔭 Recon complete! Scout has charted the territory. Map ready! 🗺️

[github-actions]

🔍 Scout Research Report

Triggered by @github-actions[bot] · /scout history=full

Executive Summary

Token usage in the rolling audit sample roughly doubled in the back half of May (~70.4M → 138.0M tokens, +96%, May 21 → May 29) while the per-report run count stayed flat (~95–98). So the trend is not more runs — it is (a) more distinct agentic workflows active in the sample (≈36 → ≈68) and (b) higher tokens per run (~0.73M → ~1.45M avg, ~2×). The root cause is workflow proliferation: 48 new agentic workflows were added in May (237 .md workflows total now), several of them individually very heavy (single runs of 5–8.7M tokens), and many of May's new workflows appear directly in the top-consumer list.

Trend (reconstructed from the agentic-token-audit report series)

Report date	Total tokens (last ~100 runs)	Runs	Active wf in sample	Avg tokens/run
2026-05-21 (#33750)	70,350,368	97	36	~725K
2026-05-28 (#35471)	117,370,174	98	68	~1.20M
2026-05-29 (#35698)	138,000,015	95	66	~1.45M

⚠️ The audit samples the most recent ~100 runs, not a true calendar window. So totals reflect recent intensity, and the reported day-over-day swing (e.g. +17.6%) is partly sampling noise from which heavy workflows happened to land in the window. Cost is $0.00 because cost data isn't populated.

Attribution — May-introduced workflows dominate the top consumers

Cross-referencing the May 29 top-consumer table against git log --diff-filter=A on .github/workflows/*.md, many of the biggest consumers were created in May 2026:

Workflow (top consumer)	Added	Tokens (30d)	Trigger
PR Sous Chef	2026-05-13	15.70M (5 runs)	per-PR
Daily Grafana/OTel Instrumentation Advisor	2026-05-04	5.51M	daily
Daily AgentRx Trace Optimizer	2026-05-12	5.19M	daily
Matt Pocock Skills Reviewer	2026-05-04	3.19M (5 runs)	per-PR
PR Code Quality Reviewer	2026-05-06	2.90M (5 runs)	per-PR
daily-experiment-report	2026-05-01	2.52M	daily
Chaos PR Bundle Fuzzer	2026-05-14	1.63M	per-PR
Outcome Collector	2026-05-12	1.55M	per-PR
OTLP Data Quality Validator	2026-05-20	0.57M	daily

Pre-existing heavy hitters (Safe Output Health Monitor 8.7M, Go Logger Enhancement 8.5M, Static Analysis Report 5.8M) are one-off/large-context runs — high per-run, but not the growth driver. The growth driver is breadth: a steady ~1.5–2 new agentic workflows/day throughout May.

Workflow additions over time & May daily cadence

New agentic .md workflows added per month: 2025-08 → 22, 09 → 33, 10 → 103, 11 → 85, 12 → 53, 2026-01 → 65, 02 → 34, 03 → 30, 04 → 59, 05 → 48.

May was a near-daily drumbeat of additions (3 on 05-01, 4 on 05-12, 4 on 05-14, 3 on 05-21, etc.). A large share are daily-* scheduled or pr-* per-PR workflows, so each addition adds recurring token load, not a one-time cost. Per-PR reviewers (PR Sous Chef, PR Code Quality Reviewer, Matt Pocock Skills Reviewer) each logged 5 runs — they recur with PR activity.

Recommendations

• Apply agentic-token-optimizer to the top 5 consumers — start with PR Sous Chef (15.7M), the OTel/AgentRx daily optimizers, and the per-PR reviewers; these are where per-run cost is concentrated.
• Gate the per-PR heavy reviewers (PR Sous Chef + PR Code Quality Reviewer + Matt Pocock Skills Reviewer overlap in scope) — consolidate or add path/label filters to cut redundant 3M+ runs per PR.
• Consolidate the proliferating daily-*-optimizer family — there are many overlapping optimizer/advisor workflows; an inventory + merge would reduce both breadth and scheduled load.
• Fix the audit metric so the trend is comparable: switch from "last ~100 runs" sampling to a true calendar window, and populate cost data — the current sampling makes day-over-day deltas noisy.
• Add per-run token budgets/caps for optimizer / deep-research / meta-orchestrator workflows that legitimately read large contexts (the 5–8.7M single-run outliers).

Key Sources

• Token Usage Audit reports: #35698 (5/29), #35471 (5/28), #33750 (5/21)
• Trend plots: linked in #35698
• Git history: .github/workflows/*.md additions (full history, --diff-filter=A)

Suggested Next Steps

1. Run agentic-token-optimizer on PR Sous Chef and the OTel/AgentRx daily advisors.
2. Review the 48 May-added workflows for necessity / overlap; deprecate or merge duplicates.
3. Harden the audit metric (calendar window + cost data) before the next trend review.

🔭 Intelligence gathered by Scout · opus48 952.3K · ◷