From fa208bc180b7d0ebc6002d513670f14688f3fc87 Mon Sep 17 00:00:00 2001 From: Stephen Belanger Date: Tue, 11 Aug 2026 02:25:07 +0800 Subject: [PATCH] Add coding agent integration skills --- .../skills/add-coding-agent-capture/SKILL.md | 37 ++++++++++++++ .../agents/openai.yaml | 4 ++ .../skills/add-coding-agent-import/SKILL.md | 35 ++++++++++++++ .../agents/openai.yaml | 4 ++ .../add-coding-agent-integration/SKILL.md | 48 +++++++++++++++++++ .../agents/openai.yaml | 4 ++ .agents/skills/add-coding-agent-run/SKILL.md | 32 +++++++++++++ .../add-coding-agent-run/agents/openai.yaml | 4 ++ .../skills/add-coding-agent-setup/SKILL.md | 30 ++++++++++++ .../add-coding-agent-setup/agents/openai.yaml | 4 ++ .../add-coding-agent-translator/SKILL.md | 35 ++++++++++++++ .../agents/openai.yaml | 4 ++ .../ship-coding-agent-integration/SKILL.md | 32 +++++++++++++ .../agents/openai.yaml | 4 ++ .../test-coding-agent-integration/SKILL.md | 33 +++++++++++++ .../agents/openai.yaml | 4 ++ AGENTS.md | 8 ++++ 17 files changed, 322 insertions(+) create mode 100644 .agents/skills/add-coding-agent-capture/SKILL.md create mode 100644 .agents/skills/add-coding-agent-capture/agents/openai.yaml create mode 100644 .agents/skills/add-coding-agent-import/SKILL.md create mode 100644 .agents/skills/add-coding-agent-import/agents/openai.yaml create mode 100644 .agents/skills/add-coding-agent-integration/SKILL.md create mode 100644 .agents/skills/add-coding-agent-integration/agents/openai.yaml create mode 100644 .agents/skills/add-coding-agent-run/SKILL.md create mode 100644 .agents/skills/add-coding-agent-run/agents/openai.yaml create mode 100644 .agents/skills/add-coding-agent-setup/SKILL.md create mode 100644 .agents/skills/add-coding-agent-setup/agents/openai.yaml create mode 100644 .agents/skills/add-coding-agent-translator/SKILL.md create mode 100644 .agents/skills/add-coding-agent-translator/agents/openai.yaml create mode 100644 .agents/skills/ship-coding-agent-integration/SKILL.md create mode 100644 .agents/skills/ship-coding-agent-integration/agents/openai.yaml create mode 100644 .agents/skills/test-coding-agent-integration/SKILL.md create mode 100644 .agents/skills/test-coding-agent-integration/agents/openai.yaml diff --git a/.agents/skills/add-coding-agent-capture/SKILL.md b/.agents/skills/add-coding-agent-capture/SKILL.md new file mode 100644 index 0000000..d1e34d7 --- /dev/null +++ b/.agents/skills/add-coding-agent-capture/SKILL.md @@ -0,0 +1,37 @@ +--- +name: add-coding-agent-capture +description: Add or review live coding-agent event capture through blocking command hooks or an in-process plugin such as a JavaScript adapter. Use when forwarding native lifecycle, model, tool, permission, or subagent events into the Braintrust daemon, or when fixing ordering, failure, or flush behavior in an adapter. +--- + +# Add coding-agent event capture + +Confirm the canonical source identity and production translator exist. If not, +use `add-coding-agent-translator` first or include it earlier in the same plan. + +## Choose the capture path + +- Prefer blocking command hooks when the agent offers ordered lifecycle hooks. +- Prefer the shared long-lived daemon client for an in-process plugin API. +- Combine hooks with transcript paths only when hooks omit data required by the + translator; keep trace construction out of the adapter. + +## Implement a thin adapter + +- Forward raw native payloads with source, session, event, capture timestamp, + agent version, and adapter version. Add cwd or worktree only when supplied + separately by the agent API. +- Preserve per-session ordering. Await daemon acknowledgement for blocking + hooks; serialize requests on a long-lived connection for in-process plugins. +- Flush on terminal lifecycle events and close persistent clients cleanly. +- Fail open with bounded diagnostics when the daemon, host CLI, or network is + unavailable. Never break an agent turn because tracing failed. +- Forward only non-secret route selections. Never handle Braintrust credentials + or send spans directly to Braintrust. +- Capture every event required by the translator and quote commands safely on + every supported platform. + +## Verify completion + +Test hook or plugin registration, payload mapping, ordering, reconnection, +failure behavior, terminal flush, configuration precedence, disablement, and +the exact source identity accepted by the production translator. diff --git a/.agents/skills/add-coding-agent-capture/agents/openai.yaml b/.agents/skills/add-coding-agent-capture/agents/openai.yaml new file mode 100644 index 0000000..f39994e --- /dev/null +++ b/.agents/skills/add-coding-agent-capture/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Add Agent Capture" + short_description: "Forward agent events through hooks or plugins" + default_prompt: "Use $add-coding-agent-capture to add daemon event capture for a coding agent." diff --git a/.agents/skills/add-coding-agent-import/SKILL.md b/.agents/skills/add-coding-agent-import/SKILL.md new file mode 100644 index 0000000..804faa5 --- /dev/null +++ b/.agents/skills/add-coding-agent-import/SKILL.md @@ -0,0 +1,35 @@ +--- +name: add-coding-agent-import +description: Add or review historical transcript import and live attach support for a coding agent, including session lookup, native parsing, synthetic lifecycle envelopes, shared translator use, incremental tailing, destination overrides, and import tests. Use when implementing or fixing import, attach, resume-session lookup, or transcript-following behavior. +--- + +# Add coding-agent transcript import + +Require a production translator and a viable native transcript containing the +facts to import. If either is missing, use `add-coding-agent-translator` or report +the fidelity blocker before implementing import. + +## Implement import and attach + +- Expose the agent, aliases, and display name through the public import command. +- Locate exactly one transcript from a validated session ID across documented + active and archived locations. Avoid unsafe symlink traversal and fail clearly + on zero or multiple matches. +- Parse records with contextual file and line errors, schema-version handling, + authoritative native session identity, and native timestamps. +- Convert transcript records into synthetic envelopes for the same source and + translator used by live capture. Never create a second trace builder or invent + facts that exist only in hooks. +- Keep historical import and live attach on the same parser, processor, + translator, and sink state. Vary only waiting and finalization. +- Tail incrementally without duplicates, tolerate partial final records, keep an + active turn open, and finalize it on interruption. +- Apply typed destination or exported parent overrides before transcript lookup + and fail fast when the host has not supplied resolved session configuration. + +## Verify completion + +Test active and archived discovery, invalid and ambiguous IDs, malformed and +empty files, one and many turns, incremental growth, no-growth polls, partial +writes, finalization, destination and parent attachment, resume behavior, and +deterministic replay through the production translator. diff --git a/.agents/skills/add-coding-agent-import/agents/openai.yaml b/.agents/skills/add-coding-agent-import/agents/openai.yaml new file mode 100644 index 0000000..c928189 --- /dev/null +++ b/.agents/skills/add-coding-agent-import/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Add Agent Import" + short_description: "Import and follow native agent transcripts" + default_prompt: "Use $add-coding-agent-import to add transcript import and attach support for a coding agent." diff --git a/.agents/skills/add-coding-agent-integration/SKILL.md b/.agents/skills/add-coding-agent-integration/SKILL.md new file mode 100644 index 0000000..86fb609 --- /dev/null +++ b/.agents/skills/add-coding-agent-integration/SKILL.md @@ -0,0 +1,48 @@ +--- +name: add-coding-agent-integration +description: Orchestrate a complete Braintrust coding-agent integration by delegating feasibility, daemon translation, event capture, setup, managed run, transcript import, verification, and shipping to the specialized repo-local skills. Use for end-to-end support for a new agent or a completeness audit of an existing integration; use the component skills directly for focused work. +--- + +# Orchestrate a coding-agent integration + +Coordinate the integration; do not implement component details from this skill. +Inspect current source, keep a requirement-to-evidence inventory, and invoke the +specialized skill for every applicable workstream. + +## Delegate the work + +1. Use `audit-agent-tracing-support` to establish whether hooks, an in-process + plugin API, transcripts, or a combination can provide sufficient data. +2. Use `add-coding-agent-translator` to establish the canonical source identity + and production daemon translator first. +3. Use `add-coding-agent-capture` to forward live native events through command + hooks or an in-process plugin. +4. Once the translator exists, delegate independently: + - persistent installation and configuration to `add-coding-agent-setup`; + - invocation-local launching to `add-coding-agent-run`; + - historical import and live transcript following to + `add-coding-agent-import`, when a viable transcript exists. +5. Use `test-coding-agent-integration` to verify every implemented surface and + the combined end-to-end path. +6. Use `ship-coding-agent-integration` only after verification evidence exists. + +Add every applicable skill to the working plan and follow that skill when its +workstream begins. Do not copy its instructions into the plan or preload skills +for workstreams that are out of scope. + +## Maintain the shared contract + +Keep one stable source identity across the translator, capture adapter, setup, +run, import, status, tests, and documentation. Keep plugins and hooks thin, +fail-open, credential-free forwarders; the daemon owns correlation, trace +construction, recovery, routing, and delivery. + +Treat a missing capability as `blocked` or explicitly `not applicable` with +evidence, never as silently complete. Do not call the integration complete +until each applicable component skill has produced implementation and test +evidence and the shipping skill has confirmed a releasable distribution path. + +## Hand off + +Report a compact matrix with one row per delegated skill: status, source +evidence, test evidence, blockers or fidelity loss, and remaining release work. diff --git a/.agents/skills/add-coding-agent-integration/agents/openai.yaml b/.agents/skills/add-coding-agent-integration/agents/openai.yaml new file mode 100644 index 0000000..d5df16a --- /dev/null +++ b/.agents/skills/add-coding-agent-integration/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Add Coding Agent Integration" + short_description: "Orchestrate all coding-agent integration layers" + default_prompt: "Use $add-coding-agent-integration to coordinate a complete coding-agent integration." diff --git a/.agents/skills/add-coding-agent-run/SKILL.md b/.agents/skills/add-coding-agent-run/SKILL.md new file mode 100644 index 0000000..eb310c5 --- /dev/null +++ b/.agents/skills/add-coding-agent-run/SKILL.md @@ -0,0 +1,32 @@ +--- +name: add-coding-agent-run +description: Add or review invocation-local managed-run support for a coding agent, including executable dispatch, temporary hook or plugin injection, route isolation, duplicate-capture suppression, process behavior, trust safety, and run-command tests. Use when implementing or fixing the command that launches an agent with tracing for one invocation. +--- + +# Add a managed coding-agent run + +Require a production translator and a capture mechanism that can be injected for +one process invocation. Use the translator or capture skill first when missing. + +## Implement managed run + +- Add the agent to public run parsing, aliases, executable dispatch, and the + canonical daemon source mapping. +- Require a resolved destination before launch and scope route overrides to the + child process tree without rewriting persistent setup. +- Inject the full capture event set using agent-specific arguments, settings, + environment, or plugin configuration. +- Suppress inherited Braintrust capture only for the managed process tree while + allowing the injected adapter to run, preventing duplicate traces. +- Preserve normal hook or plugin trust review. Never enable a global trust + bypass or authorize unrelated hooks. +- Inherit stdio, forward agent arguments verbatim, return its exit status, and + terminate and reap it on interruption. +- Quote generated commands safely across supported platforms and unusual paths. + +## Verify completion + +Test parsing, aliases, generated configuration, every injected event, route +isolation, duplicate suppression, paths with spaces and non-ASCII text, missing +executables, failure status, interruption, and the absence of transcript polling +when live hooks or plugins are the selected capture path. diff --git a/.agents/skills/add-coding-agent-run/agents/openai.yaml b/.agents/skills/add-coding-agent-run/agents/openai.yaml new file mode 100644 index 0000000..d8fb188 --- /dev/null +++ b/.agents/skills/add-coding-agent-run/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Add Agent Run" + short_description: "Launch an agent with invocation-local tracing" + default_prompt: "Use $add-coding-agent-run to add managed tracing runs for a coding agent." diff --git a/.agents/skills/add-coding-agent-setup/SKILL.md b/.agents/skills/add-coding-agent-setup/SKILL.md new file mode 100644 index 0000000..c315a75 --- /dev/null +++ b/.agents/skills/add-coding-agent-setup/SKILL.md @@ -0,0 +1,30 @@ +--- +name: add-coding-agent-setup +description: Add or review persistent setup for a coding-agent tracing integration, including public CLI exposure, plugin or hook installation, non-secret route configuration, idempotent updates, disablement, and isolated setup tests. Use when implementing or fixing the setup, install, enable, disable, or uninstall experience for an agent. +--- + +# Add persistent coding-agent setup + +Require a registered translator and an installable capture adapter. If either is +missing, use `add-coding-agent-translator` or `add-coding-agent-capture` before +considering setup complete. + +## Implement setup + +- Expose the agent through the public setup command and help surfaces. +- Resolve the selected profile, organization, and typed trace destination in the + profile-aware host before writing configuration. +- Install or enable the adapter through the agent's supported mechanism. +- Persist only non-secret route settings. Keep credentials, resolved backend + URLs, and credential leases in the host. +- Make repeated setup idempotent and atomic. Preserve unrelated plugins, hooks, + settings, and user configuration; avoid duplicate Braintrust entries. +- Keep generated hook definitions stable when the agent uses definition-based + trust, and preserve the agent's normal review boundary. +- Provide a reversible disable or uninstall path. + +## Verify completion + +Test setup in an isolated home and configuration tree, including first install, +repeat setup, route changes, existing unrelated configuration, disablement, +missing dependencies, and a real agent launch using the installed adapter. diff --git a/.agents/skills/add-coding-agent-setup/agents/openai.yaml b/.agents/skills/add-coding-agent-setup/agents/openai.yaml new file mode 100644 index 0000000..ea485cb --- /dev/null +++ b/.agents/skills/add-coding-agent-setup/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Add Agent Setup" + short_description: "Install and configure persistent agent tracing" + default_prompt: "Use $add-coding-agent-setup to add persistent tracing setup for a coding agent." diff --git a/.agents/skills/add-coding-agent-translator/SKILL.md b/.agents/skills/add-coding-agent-translator/SKILL.md new file mode 100644 index 0000000..e7cab5e --- /dev/null +++ b/.agents/skills/add-coding-agent-translator/SKILL.md @@ -0,0 +1,35 @@ +--- +name: add-coding-agent-translator +description: Implement or review a coding agent's daemon translator, including source registration, native event correlation, trace shape, deterministic recovery, routing metadata, and translator tests. Use when adding a new agent translator or changing how an existing agent's hooks or transcripts become Braintrust spans. +--- + +# Add a coding-agent translator + +Start from official agent documentation, captured native payloads, and native +transcripts. Establish one lowercase source identity for every integration +surface and record any version-dependent schema behavior. + +## Implement the translator + +- Add one stateful translator per daemon session and register its factory in the + production registry and advertised capabilities. +- Build the supported hierarchy of session, turn, LLM, tool, permission, + compaction, and recursive subagent spans. Preserve correct parents and the + effective trace root. +- Populate native timing, inputs, outputs, model, token and cache metrics, + errors, outcomes, tags, agent and adapter versions, session identity, cwd, + and useful metadata without inventing unavailable facts. +- Honor resolved destinations, exported parent/root attachment, additional + metadata, and flush behavior. Keep internal routing data out of user metadata. +- Use shared git enrichment and deterministic span identities so repeated, + duplicate, late, and journal-replayed events converge instead of duplicating. +- Bound transcript observations to what was available when an event was + captured. Handle unknown or drifted events without panicking. +- Flush late records and close defensibly any work left open by missing events. + +## Verify completion + +Add fixture-driven tests for happy paths, failures, duplicate or missing event +pairs, multiple turns, resume, supported subagents and permissions, token +accounting, attachment, cwd changes, version variants, flush, and replay. +Confirm the production registry selects this translator for the agreed source. diff --git a/.agents/skills/add-coding-agent-translator/agents/openai.yaml b/.agents/skills/add-coding-agent-translator/agents/openai.yaml new file mode 100644 index 0000000..31f1bd2 --- /dev/null +++ b/.agents/skills/add-coding-agent-translator/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Add Agent Translator" + short_description: "Build an agent event-to-trace translator" + default_prompt: "Use $add-coding-agent-translator to implement daemon translation for a coding agent." diff --git a/.agents/skills/ship-coding-agent-integration/SKILL.md b/.agents/skills/ship-coding-agent-integration/SKILL.md new file mode 100644 index 0000000..4eacdb1 --- /dev/null +++ b/.agents/skills/ship-coding-agent-integration/SKILL.md @@ -0,0 +1,32 @@ +--- +name: ship-coding-agent-integration +description: Package and release a coding-agent tracing integration through its marketplace, package registry, or distribution repository, including manifests, builds, validation, versioning, CI, documentation, publishing automation, and installed-artifact smoke tests. Use when preparing an integration for release or auditing whether it has a complete distribution path. +--- + +# Ship a coding-agent integration + +Require verification evidence from `test-coding-agent-integration`. Do not +publish, tag, or mutate a live distribution unless the user explicitly requests +that release action. + +## Prepare the distribution + +- Integrate the agent with the repository's build, validation, and publishing + system using its supported marketplace or package format. +- Validate manifests, entrypoints, generated artifacts, package allowlists, + platform binaries, dependency compatibility, and installability. +- Ensure distributed hooks and plugins remain thin daemon forwarders without + direct Braintrust credential or ingest logic. +- Add versioning, tags, release notes, provenance, and release automation suited + to the distribution channel, including a safe dry-run path. +- Document prerequisites, setup, run, import or attach, disablement, + troubleshooting, fidelity limitations, privacy behavior, and release steps. +- Configure only the narrowly required publishing and optional smoke-test + secrets. + +## Prove release readiness + +Install the built artifact through the user-facing distribution path. Verify +setup, one live traced session, managed run, import or attach when supported, +status or permalink output, and disablement without damaging unrelated agent +configuration. Add a post-deploy smoke test where the channel permits it. diff --git a/.agents/skills/ship-coding-agent-integration/agents/openai.yaml b/.agents/skills/ship-coding-agent-integration/agents/openai.yaml new file mode 100644 index 0000000..3093a05 --- /dev/null +++ b/.agents/skills/ship-coding-agent-integration/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Ship Agent Integration" + short_description: "Package and release a coding-agent integration" + default_prompt: "Use $ship-coding-agent-integration to prepare and ship a coding-agent integration." diff --git a/.agents/skills/test-coding-agent-integration/SKILL.md b/.agents/skills/test-coding-agent-integration/SKILL.md new file mode 100644 index 0000000..e350948 --- /dev/null +++ b/.agents/skills/test-coding-agent-integration/SKILL.md @@ -0,0 +1,33 @@ +--- +name: test-coding-agent-integration +description: Test or audit a coding-agent tracing integration across translator fixtures, capture adapters, setup, managed run, transcript import, daemon recovery, real-agent execution, package installation, and CI. Use when adding verification, reviewing integration completeness, diagnosing coverage gaps, or proving that emitted events become correct Braintrust traces. +--- + +# Test a coding-agent integration + +Identify the changed integration surfaces and use their specialized skills when +their intended behavior is unclear. Derive current commands and test seams from +source rather than copying another agent's test layout. + +## Build layered evidence + +- Add translator fixtures covering trace hierarchy, content, timing, metrics, + errors, duplicates, missing events, flush, and deterministic replay. +- Test adapter mapping, ordering, failure-open behavior, reconnection, and flush. +- Test setup, managed run, and import or attach independently when supported. +- Exercise the full envelope-to-journal-to-translator-to-sink pipeline, including + routing, attachment, credential redaction, restart recovery, and status errors. +- Run a real packaged agent against deterministic mock inference and mock ingest. + Assert session, turn, LLM, tool, failure, version, provenance, and terminal + delivery semantics rather than only process success. +- Install the adapter exactly as users will, including package and peer + resolution. Add minimum and current compatibility coverage when applicable. +- Cover every claimed operating system and architecture, or document the + deliberate product limitation. + +## Validate and report + +Run the repositories' canonical formatting, linting, unit, integration, locked +dependency, package, generated-artifact, and diff checks. Explicitly execute +real-agent tests that the default runner skips and confirm they actually ran. +Report exact commands, versions, results, skipped tests, and remaining gaps. diff --git a/.agents/skills/test-coding-agent-integration/agents/openai.yaml b/.agents/skills/test-coding-agent-integration/agents/openai.yaml new file mode 100644 index 0000000..9a313f1 --- /dev/null +++ b/.agents/skills/test-coding-agent-integration/agents/openai.yaml @@ -0,0 +1,4 @@ +interface: + display_name: "Test Agent Integration" + short_description: "Verify agent tracing from events to delivered spans" + default_prompt: "Use $test-coding-agent-integration to verify a coding-agent integration end to end." diff --git a/AGENTS.md b/AGENTS.md index 1b0f057..68bf21a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -22,6 +22,14 @@ Everything an agent installs lives under `src/plugins//content/`. Plugin hooks are thin fail-open shell and Windows command launchers that invoke `bt daemon hook`; they do not contain or compile a second tracing runtime. +## Adding a coding-agent integration + +For complete integrations, use `$add-coding-agent-integration` to delegate the +tracing audit, translator, capture, setup, managed run, import/attach, testing, +and shipping work to the corresponding repo-local skills. Use a component skill +directly for focused work. Every integration uses one source identity and daemon +translator; hooks and plugins remain thin, fail-open, credential-free forwarders. + ## Local development ```bash