feat: AI eval framework for benchmarking MCP servers#2414
Merged
Conversation
- leverage kv rollup mvs - allow claude access to read only in temp dir - tweaks to sysprompt
The runs/ gitignore pattern was matching src/runs/ (source code) in addition to the intended /runs/ (data directory). Anchor the pattern so only the top-level runs/ directory is ignored, and commit the three missing source files: instrument.ts, path.ts, store.ts.
Remove files that are dev-specific or experimental scratch work: - ablation/ directory (REPORT.md, manifest.tsv) - scripts/ablation.sh, ablation-report.ts - scripts/compare-prompt-variants.sh, fast-eval.sh - MCP_IMPROVEMENTS.md Also clean up README references and ablation .gitignore patterns.
- Extract spreadTimestamp() and normalizeSeverityText() shared helpers - Collapse 14 identical phase functions into streamLogPhase() generic - Compact ground-truth programmatic checks to tuple format [id, weight, pattern, neg?] - Delete dead checkoutEventLog export, unexport internal logfmtBody/jsonEventBody
- Generalize from fixed HyperDX+ClickHouse pair to config-driven MCP registry - Add dual-slot eval setup docs for A/B branch comparison - Add baseline+challengers reporting model with delta columns - Expand README with MCP config reference, field tables, and examples - Improve viewer with comparison dashboard and drill-down - Update blinding to handle arbitrary brand terms per MCP - Add --baseline, --ch-url, --no-grade, --no-judge CLI flags
🦋 Changeset detectedLatest commit: bfc75a1 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
E2E Test Results✅ All tests passed • 197 passed • 3 skipped • 1249s
Tests ran across 4 shards in parallel. |
- Add packages/hdx-eval workspace to knip.json - Remove unused @hyperdx/common-utils dependency - Remove export from internal-only symbols (SOURCE_TRACES_TABLE, SOURCE_LOGS_TABLE, DENIED_BUILT_IN_TOOLS_BASE, CONFIG_FILENAME, loadGradedPairs, instrumentRun, HyperdxConnection, MeResponse) - Remove dead code (getScenarioGroundTruth, ensureConfigDir) - Add minor changeset for @hyperdx/hdx-eval
- Fix judge error silently penalizing combined score by 60% (grade.ts) - Fix path traversal in viewer /api/batches/:batch route (server.js) - Fix off-by-one in background operation selection (latency-spike) - Fix worker pool crash-on-error losing all in-flight results (cli.ts) - Fix claudeSpawn timer leaks on spawn error and SIGTERM escalation - Fix listRunsInBatch including .grade.json/.timing.json sidecars - Remove unused innerHTML attribute from viewer el() helper (XSS vector) - Bind viewer server to 127.0.0.1 instead of all interfaces
…idation, temp cleanup, compression, type safety
- Fix normalizeSeverityText case bug: 'FATAL' now correctly returns 'ERROR'
- Add AbortSignal.timeout to all HyperDX API and health check fetch calls
- Add identifier validation in scenarioSlug to reject unsafe characters
- Clean up temp directories after subprocess exits (leaked API keys)
- Enable ClickHouse request compression for batch inserts
- Replace per-row buildResourceAttrs with pre-built pool in noisy-signals
- Narrow groundTruth type from unknown to Record<string, unknown>
- Update AGENTS.md to list hdx-eval as sixth package
- Remove dead pickSeverity import and ScenarioOutput type alias
- Replace inline require('fs') with top-level import in cli.ts
- Add anchorTime field to EvalConfig, auto-generated and saved on first run so subsequent runs reuse the same anchor automatically - Default to skip reseed (old --no-reseed behavior); add --reseed to opt in - Add --live flag to opt out of saved anchor (wall-clock now, implies --reseed) - --anchor-time <iso> now overrides and saves to eval.config.json - Update README with new CLI flags and Anchor Time section
- Add scenarioIsSeeded() check (queries traces table for any row) - run command now checks for existing data before running; auto-seeds if the scenario tables are empty or missing - Update README to document auto-seed behavior
Lead with export HDX_DEV_SLOT in the Quick Start so eval commands connect to the correct ClickHouse instance regardless of which worktree they are run from. Replace --ch-url examples in the dual-slot seeding section with HDX_DEV_SLOT for consistency. Also clarify that .env.local must be at the monorepo root.
github-actions
Bot
added
the
review/tier-4
Contributor
🔴 Tier 4 — CriticalTouches auth, data models, config, tasks, OTel pipeline, ClickHouse, or CI/CD. Why this tier:
Review process: Deep review from a domain expert. Synchronous walkthrough may be required. Stats
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
- Escape underscores and percent signs in SQL LIKE patterns for query attribution to avoid false matches on table names - Accumulate token counts from both judge attempts when retry succeeds, fixing understated cost reporting
Matches the existing exclusion pattern in listRunsInBatch (store.ts). Without this, timing sidecars are picked up as run records and produce garbage grade files.
brandon-pereira
requested review from
a team and
wrn14897
and removed request for
a team
wrn14897
reviewed
Jun 8, 2026
wrn14897
reviewed
Jun 8, 2026
wrn14897
reviewed
Jun 8, 2026
wrn14897
reviewed
Jun 8, 2026
- Wrap post-spawn block in try/finally so rmSync always runs, even if spawn() rejects (e.g. ENOENT). Prevents API keys from leaking in /tmp/hdx-eval-*/mcp-config.json. - Change runner default model from claude-sonnet-4-6 to claude-opus-4-6.
Generators already yield in ~10K-row batches, so 5K was needlessly sub-chunking each batch into 2 inserts. 100K eliminates the split with no memory/stability downside. Benchmarked locally: ~7% faster (53.5s → 49.8s for 3.6M rows).
…rvers Spawn claude with detached:true so it gets its own process group. SIGTERM still targets claude only (it handles graceful MCP shutdown). The SIGKILL escalation now uses process.kill(-pid) to kill the entire process group, ensuring orphaned MCP server children are reaped.
Member
Author
|
@wrn14897 all feedback addressed- ready for another review :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
packages/hdx-eval— an eval framework for benchmarking AI agents against observability MCP servers. The framework generates deterministic synthetic telemetry with planted anomalies, spawns Claude Code as an SRE agent, records full trajectories, and grades answers using programmatic checks + LLM-as-judge.Key Features
What is included
packages/hdx-eval/— full eval package (CLI, harness, generators, grading, reports, viewer)packages/hdx-eval/README.md— comprehensive docs covering setup, config, scenarios, CLI reference, and scoring.opencode/commands/eval-summary.md— eval analysis skill for reviewing resultsAGENTS.md— minor addition documenting common utility locations to check before writing new functionsUsage
See
packages/hdx-eval/README.mdfor full documentation.