|
1 | | -# Development of the plugin itself |
| 1 | +# Development of the plugins |
2 | 2 |
|
3 | 3 | ## Prerequisites |
4 | 4 |
|
5 | | -- Python 3.12+ |
6 | | -- [uv](https://docs.astral.sh/uv/) package manager |
| 5 | +- Python 3.12+ and [uv](https://docs.astral.sh/uv/) for the Braintrust skill |
| 6 | + evals. |
| 7 | +- Rust for the shared tracing daemon. |
| 8 | +- `jq` for plugin manifest validation and the optional fixture recorder. |
7 | 9 |
|
8 | 10 | ## Local testing |
9 | 11 |
|
10 | | -Test a plugin without installing from marketplace: |
| 12 | +Load a plugin directly without installing it from the marketplace: |
11 | 13 |
|
12 | 14 | ```bash |
13 | | -claude --plugin-dir /path/to/thisrepo/plugins/{plugin dir here} |
14 | | -# example |
15 | | -claude --plugin-dir /path/to/thisrepo/plugins/braintrust |
| 15 | +claude --plugin-dir /path/to/repo/plugins/braintrust |
| 16 | +claude --plugin-dir /path/to/repo/plugins/trace-claude-code |
16 | 17 | ``` |
17 | 18 |
|
18 | 19 | ## Running evals |
19 | 20 |
|
20 | | -The `evals/` directory contains tests that verify the plugin works correctly (e.g., Claude generates valid SQL queries, logs data properly). |
| 21 | +The `evals/` directory verifies that Claude can use Braintrust workflows: |
21 | 22 |
|
22 | 23 | ```bash |
23 | 24 | cd evals |
24 | 25 | export BRAINTRUST_API_KEY="your-key" |
25 | | - |
26 | | -# Run all evals |
27 | 26 | uv run braintrust eval . |
28 | | - |
29 | | -# Run specific eval |
30 | | -uv run braintrust eval eval_e2e_log_fetch.py |
31 | 27 | ``` |
32 | 28 |
|
33 | | -## Pre-commit hooks |
34 | | - |
35 | | -```bash |
36 | | -# Install hooks |
37 | | -uv run pre-commit install |
| 29 | +## Testing `trace-claude-code` |
38 | 30 |
|
39 | | -# Run all hooks |
40 | | -uv run pre-commit run --all-files |
41 | | -``` |
| 31 | +The plugin contains only a fail-open `bt` hook shim. All event translation and |
| 32 | +Braintrust delivery live in the shared Rust daemon at `bt-daemon/`. |
42 | 33 |
|
43 | | -## Testing the `trace-claude-code` plugin |
| 34 | +From the monorepo root: |
44 | 35 |
|
45 | | -Bash test suite for the hook scripts. Tests run the hooks against a |
46 | | -stubbed `curl`, capture the resulting HTTP requests, and assert on the |
47 | | -inferred span tree. |
48 | | - |
49 | | -### Running |
50 | | - |
51 | | -```sh |
52 | | -# From the repo root: |
| 36 | +```bash |
| 37 | +cargo test --manifest-path bt-daemon/Cargo.toml --all-features |
| 38 | +cargo clippy --manifest-path bt-daemon/Cargo.toml --all-targets --all-features -- -D warnings |
53 | 39 | make test |
54 | | - |
55 | | -# Or run a specific test file: |
56 | | -bash plugins/trace-claude-code/test/run_tests.sh test_e2e |
57 | | -bash plugins/trace-claude-code/test/run_tests.sh test_replay test_queue |
58 | 40 | ``` |
59 | 41 |
|
60 | | -### Layout |
61 | | - |
62 | | -``` |
63 | | -plugins/trace-claude-code/test/ |
64 | | -├── helpers/ |
65 | | -│ ├── assert.sh # describe / it / assert_eq / assert_contains, color output |
66 | | -│ ├── harness.sh # setup_test_env, teardown_test_env, run_hook |
67 | | -│ ├── curl_stub.sh # curl() shell function that captures requests + returns canned responses |
68 | | -│ ├── fixtures.sh # builders for hook input JSON (fixture_session_start, etc.) |
69 | | -│ ├── span_tree.sh # all_spans, span_count_by_type, span_by_name, children_of, ... |
70 | | -│ └── replay.sh # replay_session, describe_fixture |
71 | | -├── fixtures/ |
72 | | -│ └── sessions/ # captured Claude sessions used by test_replay.sh |
73 | | -├── test_*.sh # one file per area |
74 | | -├── record_session.sh # CLI to prep a fixture directory for capturing |
75 | | -└── run_tests.sh # entry point |
76 | | -``` |
| 42 | +`bt-daemon/tests/claude_translator.rs` covers synthetic lifecycle cases and |
| 43 | +replays the immutable captured sessions under |
| 44 | +`plugins/trace-claude-code/test/fixtures/sessions/`. Add translator behavior and |
| 45 | +assertions there, not as another hook script. |
77 | 46 |
|
78 | | -### Writing a test |
| 47 | +### Capturing a fixture |
79 | 48 |
|
80 | | -Each `test_*.sh` follows this pattern: |
| 49 | +Set `BRAINTRUST_RECORD_DIR` to a new absolute directory before running Claude: |
81 | 50 |
|
82 | 51 | ```bash |
83 | | -#!/bin/bash |
84 | | -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" |
85 | | -source "$SCRIPT_DIR/helpers/assert.sh" |
86 | | -source "$SCRIPT_DIR/helpers/harness.sh" |
87 | | - |
88 | | -describe "my feature" |
89 | | - |
90 | | -t_my_test_body() { |
91 | | - # setup_test_env has already created an isolated $HOME and stubbed curl |
92 | | - stub_response_for "*/v1/project_logs/*/insert" 200 '{"row_ids":["row_1"]}' |
93 | | - |
94 | | - run_hook session_start.sh "$(fixture_session_start "s1" "/tmp/x")" |
95 | | - |
96 | | - assert_eq "$(span_count_by_type task)" "1" |
97 | | -} |
98 | | - |
99 | | -it "does the thing" t_my_test_body |
100 | | -``` |
101 | | - |
102 | | -Key conventions: |
103 | | - |
104 | | -- `describe "..."` is a section header (purely visual). |
105 | | -- `it "name" function_name` runs `function_name` between `setup_test_env` |
106 | | - and `teardown_test_env`, then prints a ✓ or ✗. |
107 | | -- Assertions (`assert_eq`, `assert_contains`, `assert_failure`, ...) record |
108 | | - failures into the current test but do **not** abort. Multiple assertions |
109 | | - per test are fine. |
110 | | -- Hooks are run synchronously in tests via `BRAINTRUST_SYNC_QUEUE=true` |
111 | | - set by `setup_test_env`. Span queue tests opt out of this when needed. |
112 | | - |
113 | | -### Capturing a real session as a test fixture |
114 | | - |
115 | | -The hooks support recording every invocation to disk when the env var |
116 | | -`BRAINTRUST_RECORD_DIR` is set. The recorded data can then be replayed |
117 | | -in a test. |
118 | | - |
119 | | -#### 1. Prepare a fixture directory |
120 | | - |
121 | | -```sh |
122 | | -plugins/trace-claude-code/test/record_session.sh my-fixture |
| 52 | +export BRAINTRUST_RECORD_DIR=/absolute/path/to/new-fixture |
| 53 | +claude --plugin-dir /path/to/plugins/trace-claude-code |
123 | 54 | ``` |
124 | 55 |
|
125 | | -This prints a `BRAINTRUST_RECORD_DIR` value pointing at |
126 | | -`test/fixtures/sessions/my-fixture/`. |
127 | | - |
128 | | -#### 2. Run Claude Code with recording on |
| 56 | +The shim appends `{ts, hook, payload}` records to `events.ndjson` and copies |
| 57 | +referenced main/subagent transcripts under `transcripts/`. Move a reviewed, |
| 58 | +credential-free capture under `test/fixtures/sessions/`, add its contract to |
| 59 | +the Rust test, and run the full daemon suite. The daemon’s normal recovery |
| 60 | +journal independently embeds transcript snapshots at lifecycle boundaries. |
129 | 61 |
|
130 | | -```sh |
131 | | -export BRAINTRUST_RECORD_DIR=/abs/path/to/test/fixtures/sessions/my-fixture |
132 | | -claude |
133 | | -# ... use Claude Code normally ... |
134 | | -``` |
135 | | - |
136 | | -While `BRAINTRUST_RECORD_DIR` is set: |
137 | | - |
138 | | -- Every hook invocation appends one NDJSON record to |
139 | | - `events.ndjson` containing `{ts, hook, payload}`. |
140 | | -- The `stop_hook` also copies the referenced transcript file into |
141 | | - `transcripts/<session_id>.jsonl`. |
142 | | - |
143 | | -You do not need to modify hook scripts or set anything else - the recorder |
144 | | -runs inside the existing hooks. |
145 | | - |
146 | | -#### 3. Inspect the fixture |
147 | | - |
148 | | -```sh |
149 | | -plugins/trace-claude-code/test/record_session.sh --describe my-fixture |
150 | | -``` |
151 | | - |
152 | | -Output: |
153 | | - |
154 | | -``` |
155 | | -Fixture: .../test/fixtures/sessions/my-fixture |
156 | | - Events: 14 |
157 | | - Hook counts: |
158 | | - post_tool_use: 8 |
159 | | - session_end: 1 |
160 | | - session_start: 1 |
161 | | - stop_hook: 3 |
162 | | - user_prompt_submit: 1 |
163 | | - Transcripts: 1 |
164 | | -``` |
165 | | - |
166 | | -#### 4. Replay it in a test |
| 62 | +## Pre-commit hooks |
167 | 63 |
|
168 | 64 | ```bash |
169 | | -t_replay_my_fixture() { |
170 | | - stub_response_for "*/v1/project_logs/*/insert" 200 '{"row_ids":["row_1"]}' |
171 | | - |
172 | | - local n |
173 | | - n=$(replay_session "$SCRIPT_DIR/fixtures/sessions/my-fixture") |
174 | | - assert_success "$?" |
175 | | - assert_eq "$n" "14" |
176 | | - |
177 | | - # Now assert on the span tree the hooks produced |
178 | | - assert_eq "$(span_count_by_type tool)" "8" |
179 | | - assert_eq "$(span_count_by_type llm)" "3" |
180 | | -} |
181 | | - |
182 | | -it "my real-world fixture produces the expected spans" t_replay_my_fixture |
| 65 | +uv run pre-commit install |
| 66 | +uv run pre-commit run --all-files |
183 | 67 | ``` |
184 | 68 |
|
185 | | -The replayer: |
186 | | - |
187 | | -- Reads `events.ndjson` line by line in order. |
188 | | -- For `stop_hook` events, rewrites `payload.transcript_path` to point at |
189 | | - the bundled transcript so the replayed hook can read it. |
190 | | -- Invokes the matching hook script via `run_hook` with the recorded |
191 | | - payload. |
192 | | - |
193 | | -#### When to use replay vs. synthetic fixtures |
194 | | - |
195 | | -- **Synthetic fixtures** (`fixture_session_start`, etc.) - fast to write, |
196 | | - test specific scenarios in isolation, no real Claude needed. |
197 | | -- **Replayed fixtures** - high-fidelity regression tests of real-world |
198 | | - interactions. Use when you want to lock in behavior on a specific |
199 | | - pattern of hooks you saw in the wild (e.g. a session with parallel |
200 | | - tool calls, or a long multi-turn conversation). |
201 | | - |
202 | | -### Span-tree queries |
203 | | - |
204 | | -The captured HTTP requests are parsed to extract the inserted spans. Available helpers: |
205 | | - |
206 | | -| Function | Returns | |
207 | | -|---|---| |
208 | | -| `all_spans` | JSON array of every span sent to any `/insert` endpoint | |
209 | | -| `span_count` | total number of spans | |
210 | | -| `span_count_by_type "tool"` | count of spans with `span_attributes.type == "tool"` | |
211 | | -| `spans_named "^Turn "` | array of spans whose name matches the regex | |
212 | | -| `span_by_name "^Turn 1$"` | first matching span (or `null`) | |
213 | | -| `span_by_type "llm"` | first span of that type | |
214 | | -| `span_by_id "..."` | span with the given `span_id` | |
215 | | -| `children_of "<span_id>"` | array of spans whose first parent is the given id | |
216 | | -| `is_child_of "<child_id>" "<parent_id>"` | exit 0 if true | |
217 | | - |
218 | | -All return JSON on stdout; combine with `jq` for further drilling. |
219 | | - |
220 | 69 | # Releasing a plugin |
221 | 70 |
|
222 | | -Releases are manual and git-driven. There are no git tags or publish automation: pushing to `main` is the release. |
223 | | - |
224 | | -## How version resolution works |
225 | | - |
226 | | -Claude Code resolves a plugin's version from the first of these that is set: |
227 | | - |
228 | | -1. `version` in the plugin's `plugins/<plugin>/.claude-plugin/plugin.json` |
229 | | -2. `version` in the plugin's entry in `.claude-plugin/marketplace.json` |
230 | | -3. The git commit SHA of the plugin's source |
| 71 | +Releases are manual and git-driven. There are no git tags or publish |
| 72 | +automation: pushing to `main` is the release. |
231 | 73 |
|
232 | | -Both plugins set `version` in their own `plugin.json`, and the marketplace entries do **not** declare a per-plugin `version`. So **each plugin's `plugin.json` is the sole authority for its version**, and bumping it is what triggers updates for users. |
| 74 | +Claude Code resolves a plugin version from the first available source: |
233 | 75 |
|
234 | | -The top-level `version` field in `marketplace.json` is just marketplace-manifest metadata. It does **not** gate plugin updates. |
| 76 | +1. `version` in `plugins/<plugin>/.claude-plugin/plugin.json` |
| 77 | +2. `version` in its marketplace entry |
| 78 | +3. the source commit SHA |
235 | 79 |
|
236 | | -> [!WARNING] |
237 | | -> Do not add a `version` field to a plugin's entry in `marketplace.json`. The `plugin.json` value always wins silently, so a stale marketplace version can mask the real one. Keep the version in `plugin.json` only. |
| 80 | +Each plugin’s `plugin.json` is authoritative. Do not add a per-plugin version |
| 81 | +to `marketplace.json`; a stale duplicate can mask the real version. |
238 | 82 |
|
239 | | -## Release steps |
| 83 | +Release steps: |
240 | 84 |
|
241 | | -1. Bump `version` in the plugin's manifest: |
242 | | - - `plugins/braintrust/.claude-plugin/plugin.json`, or |
243 | | - - `plugins/trace-claude-code/.claude-plugin/plugin.json` |
244 | | -2. (Optional) Bump the top-level `version` in `.claude-plugin/marketplace.json` for bookkeeping. This is cosmetic and does not affect whether users receive the update. |
245 | | -3. Commit and push to `main` (via PR). |
246 | | -4. Users update with: `claude plugin marketplace update braintrust-claude-plugin` |
| 85 | +1. Bump the plugin’s `.claude-plugin/plugin.json` version. |
| 86 | +2. Optionally bump the marketplace manifest’s top-level bookkeeping version. |
| 87 | +3. Commit and merge through a PR. |
| 88 | +4. Users update with |
| 89 | + `claude plugin marketplace update braintrust-claude-plugin`. |
0 commit comments