Skip to content

docs: Add targeted snapshot test guidance #10

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cameroncooke wants to merge 1 commit into
base: main
Choose a base branch
from
Open

docs: Add targeted snapshot test guidance #10

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions app/docs/_content/testing.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,47 @@ npm run test:snapshots:update 2>&1 | tee /tmp/snapshot-update.txt
If a snapshot changes unexpectedly, assume the code is wrong before blaming the fixture. Review every fixture diff before committing it.
</Callout>

### Adding a test case

Edit the suite file in `src/snapshot-tests/suites/` that matches the tool category, for example `device-suite.ts`. Add an `it(...)` block inside the relevant `describe(...)` group:

```ts
it('success', async () => {
const { text, isError } = await harness.invoke('device', 'list', {});
expect(isError).toBe(false);
expectFixture(text, 'list--success');
});
```

The second argument to `expectFixture` is the scenario name. It maps to fixture files under `src/snapshot-tests/__fixtures__/`:

| Runtime | Path pattern |
|---------|--------------|
| `mcp` | `mcp/{workflow}/{scenario}.txt` |
| `cli` | `cli/{workflow}/{scenario}.txt` |
| `json` | `json/{workflow}/{scenario}.json` |

Generate the initial fixture by running the suite with `UPDATE_SNAPSHOTS=1` (see below).

### Targeting one test

Pass the test file path and use `-t` to filter by test name:

```shell
npx vitest run --config vitest.snapshot.config.ts src/snapshot-tests/__tests__/device.snapshot.test.ts
npx vitest run --config vitest.snapshot.config.ts src/snapshot-tests/__tests__/device.snapshot.test.ts -t "list"
```

### Updating a targeted test

Combine `UPDATE_SNAPSHOTS=1` with a file filter and `-t` to regenerate only the matched fixtures:

```shell
UPDATE_SNAPSHOTS=1 npx vitest run --config vitest.snapshot.config.ts src/snapshot-tests/__tests__/device.snapshot.test.ts -t "list success"
Copy link
Copy Markdown

@sentry sentry Bot Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: The documented Vitest filter command -t "list success" may not work if tests are nested, as Vitest uses > as a separator, not a space.
Severity: MEDIUM

Suggested Fix

Verify the actual test suite structure in the getsentry/XcodeBuildMCP repository. If tests are nested, update the command in the documentation to use the correct separator, for example: -t "list > success". Alternatively, clarify the documentation by showing the exact describe/it structure that results in the documented command working correctly.

Prompt for AI Agent 
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: app/docs/_content/testing.mdx#L147

Potential issue: The documentation for running snapshot tests provides a Vitest filter
command, `-t "list success"`, to run a specific test. However, Vitest's official
documentation specifies that nested `describe` and `it` blocks are separated by a `>`
character in the full test name (e.g., `"list > success"`). The provided command uses a
space, which would only work if the test name was flat, like `it('list success', ...)`.
Because the actual test suite structure is not shown and nesting is a common pattern,
the documented command is likely to silently fail by matching zero tests, causing
confusion for contributors trying to use this feature.

Did we get this right? 👍 / 👎 to inform future reviews.

```

This writes only the fixture files for the matched tests. All other fixtures are left untouched.

JSON fixtures must also validate against the canonical structured-output schemas:

```shell
Expand Down
Loading