Skip to content

Commit fce3233

Browse files
committed
Switch coding-agent plugins to the Rust daemon
Replace the legacy Codex TypeScript runtime and Claude shell state machine with fail-open launchers for the shared bt daemon. This reapplies the exact plugin source migration split out of the daemon infrastructure PR and is intentionally stacked until a daemon-capable bt release is available. Signed-off-by: Stephen Belanger <stephen.belanger@braintrustdata.com>
1 parent 709f7bd commit fce3233

114 files changed

Lines changed: 534 additions & 24458 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 46 additions & 203 deletions
Original file line numberDiff line numberDiff line change
@@ -1,246 +1,89 @@
1-
# Development of the plugin itself
1+
# Development of the plugins
22

33
## Prerequisites
44

5-
- Python 3.12+
6-
- [uv](https://docs.astral.sh/uv/) package manager
5+
- Python 3.12+ and [uv](https://docs.astral.sh/uv/) for the Braintrust skill
6+
evals.
7+
- Rust for the shared tracing daemon.
8+
- `jq` for plugin manifest validation and the optional fixture recorder.
79

810
## Local testing
911

10-
Test a plugin without installing from marketplace:
12+
Load a plugin directly without installing it from the marketplace:
1113

1214
```bash
13-
claude --plugin-dir /path/to/thisrepo/plugins/{plugin dir here}
14-
# example
15-
claude --plugin-dir /path/to/thisrepo/plugins/braintrust
15+
claude --plugin-dir /path/to/repo/plugins/braintrust
16+
claude --plugin-dir /path/to/repo/plugins/trace-claude-code
1617
```
1718

1819
## Running evals
1920

20-
The `evals/` directory contains tests that verify the plugin works correctly (e.g., Claude generates valid SQL queries, logs data properly).
21+
The `evals/` directory verifies that Claude can use Braintrust workflows:
2122

2223
```bash
2324
cd evals
2425
export BRAINTRUST_API_KEY="your-key"
25-
26-
# Run all evals
2726
uv run braintrust eval .
28-
29-
# Run specific eval
30-
uv run braintrust eval eval_e2e_log_fetch.py
3127
```
3228

33-
## Pre-commit hooks
34-
35-
```bash
36-
# Install hooks
37-
uv run pre-commit install
29+
## Testing `trace-claude-code`
3830

39-
# Run all hooks
40-
uv run pre-commit run --all-files
41-
```
31+
The plugin contains only a fail-open `bt` hook shim. All event translation and
32+
Braintrust delivery live in the shared Rust daemon at `bt-daemon/`.
4233

43-
## Testing the `trace-claude-code` plugin
34+
From the monorepo root:
4435

45-
Bash test suite for the hook scripts. Tests run the hooks against a
46-
stubbed `curl`, capture the resulting HTTP requests, and assert on the
47-
inferred span tree.
48-
49-
### Running
50-
51-
```sh
52-
# From the repo root:
36+
```bash
37+
cargo test --manifest-path bt-daemon/Cargo.toml --all-features
38+
cargo clippy --manifest-path bt-daemon/Cargo.toml --all-targets --all-features -- -D warnings
5339
make test
54-
55-
# Or run a specific test file:
56-
bash plugins/trace-claude-code/test/run_tests.sh test_e2e
57-
bash plugins/trace-claude-code/test/run_tests.sh test_replay test_queue
5840
```
5941

60-
### Layout
61-
62-
```
63-
plugins/trace-claude-code/test/
64-
├── helpers/
65-
│ ├── assert.sh # describe / it / assert_eq / assert_contains, color output
66-
│ ├── harness.sh # setup_test_env, teardown_test_env, run_hook
67-
│ ├── curl_stub.sh # curl() shell function that captures requests + returns canned responses
68-
│ ├── fixtures.sh # builders for hook input JSON (fixture_session_start, etc.)
69-
│ ├── span_tree.sh # all_spans, span_count_by_type, span_by_name, children_of, ...
70-
│ └── replay.sh # replay_session, describe_fixture
71-
├── fixtures/
72-
│ └── sessions/ # captured Claude sessions used by test_replay.sh
73-
├── test_*.sh # one file per area
74-
├── record_session.sh # CLI to prep a fixture directory for capturing
75-
└── run_tests.sh # entry point
76-
```
42+
`bt-daemon/tests/claude_translator.rs` covers synthetic lifecycle cases and
43+
replays the immutable captured sessions under
44+
`plugins/trace-claude-code/test/fixtures/sessions/`. Add translator behavior and
45+
assertions there, not as another hook script.
7746

78-
### Writing a test
47+
### Capturing a fixture
7948

80-
Each `test_*.sh` follows this pattern:
49+
Set `BRAINTRUST_RECORD_DIR` to a new absolute directory before running Claude:
8150

8251
```bash
83-
#!/bin/bash
84-
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
85-
source "$SCRIPT_DIR/helpers/assert.sh"
86-
source "$SCRIPT_DIR/helpers/harness.sh"
87-
88-
describe "my feature"
89-
90-
t_my_test_body() {
91-
# setup_test_env has already created an isolated $HOME and stubbed curl
92-
stub_response_for "*/v1/project_logs/*/insert" 200 '{"row_ids":["row_1"]}'
93-
94-
run_hook session_start.sh "$(fixture_session_start "s1" "/tmp/x")"
95-
96-
assert_eq "$(span_count_by_type task)" "1"
97-
}
98-
99-
it "does the thing" t_my_test_body
100-
```
101-
102-
Key conventions:
103-
104-
- `describe "..."` is a section header (purely visual).
105-
- `it "name" function_name` runs `function_name` between `setup_test_env`
106-
and `teardown_test_env`, then prints a ✓ or ✗.
107-
- Assertions (`assert_eq`, `assert_contains`, `assert_failure`, ...) record
108-
failures into the current test but do **not** abort. Multiple assertions
109-
per test are fine.
110-
- Hooks are run synchronously in tests via `BRAINTRUST_SYNC_QUEUE=true`
111-
set by `setup_test_env`. Span queue tests opt out of this when needed.
112-
113-
### Capturing a real session as a test fixture
114-
115-
The hooks support recording every invocation to disk when the env var
116-
`BRAINTRUST_RECORD_DIR` is set. The recorded data can then be replayed
117-
in a test.
118-
119-
#### 1. Prepare a fixture directory
120-
121-
```sh
122-
plugins/trace-claude-code/test/record_session.sh my-fixture
52+
export BRAINTRUST_RECORD_DIR=/absolute/path/to/new-fixture
53+
claude --plugin-dir /path/to/plugins/trace-claude-code
12354
```
12455

125-
This prints a `BRAINTRUST_RECORD_DIR` value pointing at
126-
`test/fixtures/sessions/my-fixture/`.
127-
128-
#### 2. Run Claude Code with recording on
56+
The shim appends `{ts, hook, payload}` records to `events.ndjson` and copies
57+
referenced main/subagent transcripts under `transcripts/`. Move a reviewed,
58+
credential-free capture under `test/fixtures/sessions/`, add its contract to
59+
the Rust test, and run the full daemon suite. The daemon’s normal recovery
60+
journal independently embeds transcript snapshots at lifecycle boundaries.
12961

130-
```sh
131-
export BRAINTRUST_RECORD_DIR=/abs/path/to/test/fixtures/sessions/my-fixture
132-
claude
133-
# ... use Claude Code normally ...
134-
```
135-
136-
While `BRAINTRUST_RECORD_DIR` is set:
137-
138-
- Every hook invocation appends one NDJSON record to
139-
`events.ndjson` containing `{ts, hook, payload}`.
140-
- The `stop_hook` also copies the referenced transcript file into
141-
`transcripts/<session_id>.jsonl`.
142-
143-
You do not need to modify hook scripts or set anything else - the recorder
144-
runs inside the existing hooks.
145-
146-
#### 3. Inspect the fixture
147-
148-
```sh
149-
plugins/trace-claude-code/test/record_session.sh --describe my-fixture
150-
```
151-
152-
Output:
153-
154-
```
155-
Fixture: .../test/fixtures/sessions/my-fixture
156-
Events: 14
157-
Hook counts:
158-
post_tool_use: 8
159-
session_end: 1
160-
session_start: 1
161-
stop_hook: 3
162-
user_prompt_submit: 1
163-
Transcripts: 1
164-
```
165-
166-
#### 4. Replay it in a test
62+
## Pre-commit hooks
16763

16864
```bash
169-
t_replay_my_fixture() {
170-
stub_response_for "*/v1/project_logs/*/insert" 200 '{"row_ids":["row_1"]}'
171-
172-
local n
173-
n=$(replay_session "$SCRIPT_DIR/fixtures/sessions/my-fixture")
174-
assert_success "$?"
175-
assert_eq "$n" "14"
176-
177-
# Now assert on the span tree the hooks produced
178-
assert_eq "$(span_count_by_type tool)" "8"
179-
assert_eq "$(span_count_by_type llm)" "3"
180-
}
181-
182-
it "my real-world fixture produces the expected spans" t_replay_my_fixture
65+
uv run pre-commit install
66+
uv run pre-commit run --all-files
18367
```
18468

185-
The replayer:
186-
187-
- Reads `events.ndjson` line by line in order.
188-
- For `stop_hook` events, rewrites `payload.transcript_path` to point at
189-
the bundled transcript so the replayed hook can read it.
190-
- Invokes the matching hook script via `run_hook` with the recorded
191-
payload.
192-
193-
#### When to use replay vs. synthetic fixtures
194-
195-
- **Synthetic fixtures** (`fixture_session_start`, etc.) - fast to write,
196-
test specific scenarios in isolation, no real Claude needed.
197-
- **Replayed fixtures** - high-fidelity regression tests of real-world
198-
interactions. Use when you want to lock in behavior on a specific
199-
pattern of hooks you saw in the wild (e.g. a session with parallel
200-
tool calls, or a long multi-turn conversation).
201-
202-
### Span-tree queries
203-
204-
The captured HTTP requests are parsed to extract the inserted spans. Available helpers:
205-
206-
| Function | Returns |
207-
|---|---|
208-
| `all_spans` | JSON array of every span sent to any `/insert` endpoint |
209-
| `span_count` | total number of spans |
210-
| `span_count_by_type "tool"` | count of spans with `span_attributes.type == "tool"` |
211-
| `spans_named "^Turn "` | array of spans whose name matches the regex |
212-
| `span_by_name "^Turn 1$"` | first matching span (or `null`) |
213-
| `span_by_type "llm"` | first span of that type |
214-
| `span_by_id "..."` | span with the given `span_id` |
215-
| `children_of "<span_id>"` | array of spans whose first parent is the given id |
216-
| `is_child_of "<child_id>" "<parent_id>"` | exit 0 if true |
217-
218-
All return JSON on stdout; combine with `jq` for further drilling.
219-
22069
# Releasing a plugin
22170

222-
Releases are manual and git-driven. There are no git tags or publish automation: pushing to `main` is the release.
223-
224-
## How version resolution works
225-
226-
Claude Code resolves a plugin's version from the first of these that is set:
227-
228-
1. `version` in the plugin's `plugins/<plugin>/.claude-plugin/plugin.json`
229-
2. `version` in the plugin's entry in `.claude-plugin/marketplace.json`
230-
3. The git commit SHA of the plugin's source
71+
Releases are manual and git-driven. There are no git tags or publish
72+
automation: pushing to `main` is the release.
23173

232-
Both plugins set `version` in their own `plugin.json`, and the marketplace entries do **not** declare a per-plugin `version`. So **each plugin's `plugin.json` is the sole authority for its version**, and bumping it is what triggers updates for users.
74+
Claude Code resolves a plugin version from the first available source:
23375

234-
The top-level `version` field in `marketplace.json` is just marketplace-manifest metadata. It does **not** gate plugin updates.
76+
1. `version` in `plugins/<plugin>/.claude-plugin/plugin.json`
77+
2. `version` in its marketplace entry
78+
3. the source commit SHA
23579

236-
> [!WARNING]
237-
> Do not add a `version` field to a plugin's entry in `marketplace.json`. The `plugin.json` value always wins silently, so a stale marketplace version can mask the real one. Keep the version in `plugin.json` only.
80+
Each plugin’s `plugin.json` is authoritative. Do not add a per-plugin version
81+
to `marketplace.json`; a stale duplicate can mask the real version.
23882

239-
## Release steps
83+
Release steps:
24084

241-
1. Bump `version` in the plugin's manifest:
242-
- `plugins/braintrust/.claude-plugin/plugin.json`, or
243-
- `plugins/trace-claude-code/.claude-plugin/plugin.json`
244-
2. (Optional) Bump the top-level `version` in `.claude-plugin/marketplace.json` for bookkeeping. This is cosmetic and does not affect whether users receive the update.
245-
3. Commit and push to `main` (via PR).
246-
4. Users update with: `claude plugin marketplace update braintrust-claude-plugin`
85+
1. Bump the plugin’s `.claude-plugin/plugin.json` version.
86+
2. Optionally bump the marketplace manifest’s top-level bookkeeping version.
87+
3. Commit and merge through a PR.
88+
4. Users update with
89+
`claude plugin marketplace update braintrust-claude-plugin`.
Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
.PHONY: test test-trace-claude-code
22

3-
# Run all plugin tests
43
test: test-trace-claude-code
54

6-
# Run trace-claude-code plugin tests
75
test-trace-claude-code:
8-
@bash plugins/trace-claude-code/test/run_tests.sh
6+
@sh -n plugins/trace-claude-code/bin/claude-hook.sh
7+
@jq empty plugins/trace-claude-code/hooks/hooks.json
8+
@grep -q "'daemon','hook','--source','claude-code'" plugins/trace-claude-code/bin/claude-hook.cmd
9+
@echo "trace-claude-code shims OK"

src/plugins/claude/content/README.md

Lines changed: 30 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,8 @@ A Claude Code plugin marketplace for [Braintrust](https://braintrust.dev) integr
1010
## Prerequisites
1111

1212
- A [Braintrust account](https://braintrust.dev)
13-
- `BRAINTRUST_API_KEY` exported in your environment
13+
- The [`bt` CLI](https://bt.dev/cli/install.sh), authenticated with `bt auth login`,
14+
or `BRAINTRUST_API_KEY` exported in your environment
1415

1516
## Installation
1617

@@ -37,31 +38,45 @@ claude plugin install braintrust@braintrust-claude-plugin
3738

3839
### trace-claude-code
3940

40-
Automatically traces Claude Code conversations to Braintrust. Captures sessions, conversation turns, and tool calls as hierarchical traces.
41+
Automatically traces Claude Code conversations to Braintrust through the
42+
shared local `bt` daemon. Captures sessions, turns, model calls, tool calls,
43+
subagents, tool failures, and permission denials as hierarchical traces.
4144

4245
```bash
4346
claude plugin install trace-claude-code@braintrust-claude-plugin
4447
# run the setup script to confgure tracing
4548
$HOME/.claude/plugins/marketplaces/braintrust-claude-plugin/plugins/trace-claude-code/setup.sh
4649
```
4750

48-
Traces are sent to the `claude-code` project by default.
51+
The setup script enables tracing in the daemon's shared non-credential
52+
`config.json`. The project selected there applies to every coding-agent plugin
53+
connected to the daemon.
54+
55+
The tracing launchers live under `plugins/trace-claude-code/bin/` as
56+
`claude-hook.sh` and `claude-hook.cmd`. The Windows launcher forwards the same
57+
configuration to `bt`, whose daemon uses a local named pipe on Windows.
4958

5059
#### manual configuration
5160

52-
Instead of running `setup.sh`, you can manually edit `~/.claude/settings.json` or your project's `.claude/settings.local.json`:
61+
Instead of running `setup.sh`, create the daemon's shared config at
62+
`$BT_DAEMON_CONFIG`, or at `config.json` inside `$BT_DAEMON_DATA_DIR` (by
63+
default `~/.braintrust/state/bt-daemon/config.json` on Unix):
5364

5465
```json
5566
{
56-
"env": {
57-
"TRACE_TO_BRAINTRUST": "true",
58-
"BRAINTRUST_CC_PROJECT": "project-name-to-send-cc-traces-to",
59-
"BRAINTRUST_API_KEY": "sk-yourkey",
60-
"BRAINTRUST_DEBUG": "false"
61-
}
67+
"traceToBraintrust": true,
68+
"project": "coding-agents",
69+
"flushOnTurnEnd": false,
70+
"additionalMetadata": {"team": "platform"}
6271
}
6372
```
6473

74+
This file intentionally contains no API keys, auth tokens, credentials, or
75+
backend URLs. Those come from the authenticated `bt` CLI. The legacy
76+
`TRACE_TO_BRAINTRUST`, `BRAINTRUST_CC_PROJECT`,
77+
`BRAINTRUST_FLUSH_ON_TURN_END`, and `BRAINTRUST_ADDITIONAL_METADATA`
78+
environment variables remain fallbacks for omitted shared settings.
79+
6580
#### add claude code trace to an existing trace
6681

6782
You can attach a Claude Code session to an existing Braintrust trace by passing `CC_PARENT_SPAN_ID`:
@@ -78,11 +93,12 @@ claude --settings '{"env":{"CC_PARENT_SPAN_ID":"parent-span-id","CC_ROOT_SPAN_ID
7893

7994
The Claude Code session and all its turns/tools will appear as children of your parent span in Braintrust.
8095

81-
To attach claude code to an experiment's trace, specify CC_EXPERIMENT_ID as well:
96+
To route the session into an existing experiment instead of project logs, set
97+
`CC_EXPERIMENT_ID`.
8298

83-
```bash
84-
claude --settings '{"env":{"CC_PARENT_SPAN_ID":"parent-span-id","CC_ROOT_SPAN_ID":"root-span-id", "CC_EXPERIMENT_ID":"the-experiment-id"}}' -p "task"
85-
```
99+
Set `BRAINTRUST_RECORD_DIR` to capture native hook events and transcript
100+
snapshots for a reproducible local fixture. The daemon also embeds transcript
101+
snapshots in its redacted recovery journal at lifecycle boundaries.
86102

87103
#### token accounting
88104

src/plugins/claude/content/plugins/trace-claude-code/.claude-plugin/plugin.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
22
"name": "trace-claude-code",
33
"description": "Automatically trace Claude Code conversations to Braintrust for observability. Captures sessions, conversation turns, and tool calls as hierarchical traces.",
4-
"version": "1.5.0",
4+
"version": "2.0.0",
55
"author": {
66
"name": "Braintrust"
77
}

0 commit comments

Comments
 (0)