Support vitest 5 by GuillaumeLagrange · Pull Request #86 · CodSpeedHQ/codspeed-node

GuillaumeLagrange · 2026-06-29T07:59:11Z

No description provided.

sheremet-va · 2026-06-30T09:42:07Z

+  }
+}
+
+function patchRunBenchmarks(): void {


By the way, you can do that just in a setup file without overriding the runner. This should also make it support the browser(?)

If there is any feedback, I would love to address them on vitest side; we would also like to use codspeed after Vitest 5 is out in our own repo

@sheremet-va thanks for the heads up, currently trying out this approach.

The main pain point I am running into is that we are struggling to highjack the benchmark runner when we are in simulation/analysis mode.

When in this mode, what we used to do is highjack the whole run function, to have a very simple sequence of

warmup

manually call optimizer

start valgrind instrumentation

execute code once

stop

But I'm struggling a bit to find a good way to replicate the behavior without overriding the whole runner.
I still do not have a proper api request, nor am I sure it's the way to go for the analysis mode, becuase maybe running browser mode under valgrind is not a good idea in the first place, and the main motivation for not highjacking the whole runner is actually to be as compatible as possible with the browser mode.

Could it be possible to have a way for the plugin to hook/customize the createTinybench function? Maybe by defining some sort of abstract "benchmark backend" interface that we could fullfill, that would default to a tinybench implementation. It would also make it easier to switch away from tinybench on your end at some point.

The way tinybench is instanciated/ran does not allow for much flexibility on our end, and I cannot find a good implementation without overriding the whole runner as we did before for our analysis mode.

I'd love to exchange over this over discord/slack/a call if you want, or we can keep things here on github.

Feel free to join the discord - https://chat.vitest.dev/

I will give you the ecosystem role, we can discuss it there. I just need your nickname

I've joined and pinged on via MP, my nickname on the server is the same is github

codspeed-hq · 2026-07-01T09:07:39Z

Merging this PR will regress 2 benchmarks

⚠️

Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 3 improved benchmarks
❌ 3 (👁 1) regressed benchmarks
✅ 179 untouched benchmarks
🆕 12 new benchmarks
⏩ 53 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	WallTime	`switch 1`	144 ns	192 ns	-25%
❌	WallTime	`switch 2`	180 ns	228 ns	-21.05%
⚡	Simulation	`switch 1`	313.8 µs	100.4 µs	×3.1
⚡	Memory	`wait 500ms`	64.1 KB	23 KB	×2.8
⚡	WallTime	`short body`	1.8 µs	1.6 µs	+11.21%
🆕	WallTime	`iterative fibo 15`	N/A	900 ns	N/A
🆕	WallTime	`iterative fibo 20`	N/A	900 ns	N/A
🆕	WallTime	`recursive fibo 15`	N/A	22.4 µs	N/A
🆕	WallTime	`recursive fibo 20`	N/A	239.9 µs	N/A
🆕	Memory	`iterative fibo 15`	N/A	0 B	N/A
🆕	Memory	`iterative fibo 20`	N/A	0 B	N/A
🆕	Memory	`recursive fibo 15`	N/A	19 B	N/A
🆕	Memory	`recursive fibo 20`	N/A	19 B	N/A
🆕	Simulation	`iterative fibo 15`	N/A	17 µs	N/A
🆕	Simulation	`iterative fibo 20`	N/A	17.5 µs	N/A
🆕	Simulation	`recursive fibo 15`	N/A	153.7 µs	N/A
🆕	Simulation	`recursive fibo 20`	N/A	1.4 ms	N/A
👁	Simulation	`wait 500ms`	10.2 ms	12 ms	-14.67%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing cod-2931-prepare-compatibility-with-vitest-5 (eec7ec7) with main (1999ec9)}

53 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Vitest 5 reworked the benchmark backend: the dedicated `NodeBenchmarkRunner` and the `vitest/runners` / `vitest/suite` entrypoints are gone, benchmarks now run inside `test()` through the unified `TestRunner`, and tinybench moved to v6 (stats moved from `result.benchmark` to `task.result.latency`, `includeSamples` became `retainSamples`). Detect the installed Vitest generation and select the integration seam behind a `VitestBackend` abstraction so the rest of the plugin never inspects the version: - v3/4 keep the custom benchmark runner per instrument mode (analysis/walltime). - v5 installs instrumentation from a setup file that patches the shared `TestRunner.runBenchmarks` static. A setup file (rather than a custom `test.runner`) leaves the runner untouched for non-benchmark tests and also applies to the browser pool.

Now that the plugin's own dev dependency tracks Vitest 5, add a dedicated Vitest 4 example so the legacy (v3/4) benchmark seam keeps explicit coverage alongside the existing with-vitest-v3 example. Mirrors that example, pinning vitest ^4.1.9. Refs COD-2931 Co-Authored-By: Claude <noreply@anthropic.com>

Vitest 5 exposes a `benchmark.provider` option that hands a plugin the raw benchmark functions and their hooks directly. Wire the V5 backend to register a provider through `test.benchmark.provider` and add the provider itself, which runs analysis and walltime modes off the registrations it receives. Drop the previous V5 seam that patched the static `TestRunner.runBenchmarks` and `Bench.prototype.add` from a setup file, along with the `WeakMap` fn-capture it needed to work around tinybench v6's private `fn` field. The provider gets the fn and options as data, so none of that interception remains. The legacy (Vitest 3/4) runner path is untouched. Point the rollup build and the config-injection test at the new provider entry. Refs COD-2931

greptile-apps · 2026-07-19T18:51:01Z

Greptile Summary

This PR adds Vitest 5 support to the CodSpeed Vitest plugin. The main changes are:

A new Vitest 5 benchmark provider path.
Legacy Vitest 3/4 runner code moved under src/legacy.
Shared instrumentation and walltime result conversion helpers.
Updated Vitest dependency ranges and a Vitest 4 example project.

Confidence Score: 4/5

The new Vitest 5 path can record distinct benchmarks under one CodSpeed URI, and legacy fallback can skip instrumentation.

Multiple bench() registrations in one Vitest 5 group can share one CodSpeed benchmark identity.
Unknown Vitest version detection can route supported Vitest 3/4 runs through the v5 gating path.
The remaining changes are mostly layout, tests, examples, and shared helper refactors.

packages/vitest-plugin/src/v5/provider.ts and packages/vitest-plugin/src/vitestBackend.ts

Important Files Changed

Filename	Overview
packages/vitest-plugin/src/v5/provider.ts	Adds the Vitest 5 provider, but the current URI construction can collapse multiple registrations in one group.
packages/vitest-plugin/src/vitestBackend.ts	Adds version-specific backend selection, with a risky unknown-version fallback to the v5 path.
packages/vitest-plugin/src/instrument.ts	Adds shared analysis and walltime instrumentation helpers.
packages/vitest-plugin/src/index.ts	Delegates plugin apply and config behavior to the selected Vitest backend.
packages/vitest-plugin/src/legacy/walltime.ts	Moves the legacy walltime runner to the new legacy layout.
packages/vitest-plugin/src/globalSetup.ts	Makes setup and teardown idempotent for repeated Vitest 5 setup calls.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Vitest run] --> B{Detected Vitest major}
  B -->|3 or 4| C[Legacy backend]
  C --> D[legacy analysis or walltime runner]
  B -->|5 or newer| E[V5 backend]
  E --> F{benchmark enabled}
  F -->|no| G[default Vitest behavior]
  F -->|yes| H[v5 benchmark provider]
  H --> I{CodSpeed mode}
  I -->|analysis or simulation| J[run one measured pass]
  I -->|walltime| K[run tinybench with hooks]
  J --> L[CodSpeed benchmark result]
  K --> L

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
  A[Vitest run] --> B{Detected Vitest major}
  B -->|3 or 4| C[Legacy backend]
  C --> D[legacy analysis or walltime runner]
  B -->|5 or newer| E[V5 backend]
  E --> F{benchmark enabled}
  F -->|no| G[default Vitest behavior]
  F -->|yes| H[v5 benchmark provider]
  H --> I{CodSpeed mode}
  I -->|analysis or simulation| J[run one measured pass]
  I -->|walltime| K[run tinybench with hooks]
  J --> L[CodSpeed benchmark result]
  K --> L

Comments Outside Diff (2)

packages/vitest-plugin/src/v5/provider.ts, line 1778 (link)

Group URI Drops Benchmark Name

When a Vitest 5 test registers multiple bench(name, ...) entries, this provider builds one URI from the group and reuses it for every registration. The legacy paths append the benchmark task name, so the new path can attribute distinct measurements to the same CodSpeed benchmark and merge or overwrite their results.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/vitest-plugin/src/v5/provider.ts
Line: 1778

Comment:
**Group URI Drops Benchmark Name**

When a Vitest 5 test registers multiple `bench(name, ...)` entries, this provider builds one URI from the group and reuses it for every registration. The legacy paths append the benchmark task name, so the new path can attribute distinct measurements to the same CodSpeed benchmark and merge or overwrite their results.

How can I resolve this? If you propose a fix, please make it concise.

packages/vitest-plugin/src/vitestBackend.ts, line 1910-1911 (link)

Unknown Version Selects V5

When Vitest resolution from process.cwd() fails for a supported Vitest 3/4 project, this fallback selects the v5 backend. A v3/v4 benchmark run then has no test.benchmark.enabled flag, so config() returns undefined and the legacy runner is never installed, leaving CodSpeed runs uninstrumented without an error.

Prompt To Fix With AI

This is a comment left during a code review.
Path: packages/vitest-plugin/src/vitestBackend.ts
Line: 1910-1911

Comment:
**Unknown Version Selects V5**

When Vitest resolution from `process.cwd()` fails for a supported Vitest 3/4 project, this fallback selects the v5 backend. A v3/v4 benchmark run then has no `test.benchmark.enabled` flag, so `config()` returns `undefined` and the legacy runner is never installed, leaving CodSpeed runs uninstrumented without an error.

How can I resolve this? If you propose a fix, please make it concise.

Prompt To Fix All With AI

Fix the following 3 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 3
packages/vitest-plugin/src/v5/provider.ts:1778
**Group URI Drops Benchmark Name**

When a Vitest 5 test registers multiple `bench(name, ...)` entries, this provider builds one URI from the group and reuses it for every registration. The legacy paths append the benchmark task name, so the new path can attribute distinct measurements to the same CodSpeed benchmark and merge or overwrite their results.

### Issue 2 of 3
packages/vitest-plugin/src/vitestBackend.ts:1910-1911
**Unknown Version Selects V5**

When Vitest resolution from `process.cwd()` fails for a supported Vitest 3/4 project, this fallback selects the v5 backend. A v3/v4 benchmark run then has no `test.benchmark.enabled` flag, so `config()` returns `undefined` and the legacy runner is never installed, leaving CodSpeed runs uninstrumented without an error.

### Issue 3 of 3
packages/vitest-plugin/src/globalSetup.ts:15
**Untracked TODO Comment**

This new TODO has no Linear issue reference, which violates the repository rule for TODO comments. Please change it to the `TODO(COD-XXX): ...` form or remove it.

```suggestion
// TODO(COD-XXX): Check if this can be avoided
```

_{Reviews (1): Last reviewed commit: "feat(vitest-plugin): use the vitest 5 be..." | Re-trigger Greptile}

greptile-apps · 2026-07-19T18:51:06Z

+let setupHappened = false;
 let teardownHappened = false;

+// TODO: Check if this can be avoided


Untracked TODO Comment

This new TODO has no Linear issue reference, which violates the repository rule for TODO comments. Please change it to the TODO(COD-XXX): ... form or remove it.

Suggested change

// TODO: Check if this can be avoided

// TODO(COD-XXX): Check if this can be avoided

Rule Used: Every TODO comment must reference a Linear issue: ... (source)

Prompt To Fix With AI

This is a comment left during a code review. Path: packages/vitest-plugin/src/globalSetup.ts Line: 15 Comment: **Untracked TODO Comment** This new TODO has no Linear issue reference, which violates the repository rule for TODO comments. Please change it to the `TODO(COD-XXX): ...` form or remove it. ```suggestion // TODO(COD-XXX): Check if this can be avoided ``` **Rule Used:** Every TODO comment must reference a Linear issue: ... ([source](https://app.greptile.com/codspeed/-/custom-context?memory=65193bc9-f65b-477d-9521-104b5aac5931)) How can I resolve this? If you propose a fix, please make it concise.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

GuillaumeLagrange changed the title ~~Cod 2931 prepare compatibility with vitest 5~~ Support vitest 5 Jun 29, 2026

sheremet-va reviewed Jun 30, 2026

View reviewed changes

GuillaumeLagrange force-pushed the cod-2931-prepare-compatibility-with-vitest-5 branch 5 times, most recently from 25ef927 to e04f0da Compare July 1, 2026 08:59

GuillaumeLagrange and others added 2 commits July 1, 2026 11:07

GuillaumeLagrange force-pushed the cod-2931-prepare-compatibility-with-vitest-5 branch from 6791750 to f511dee Compare July 1, 2026 11:03

--wip-- [skip ci]

c3a356b

GuillaumeLagrange force-pushed the cod-2931-prepare-compatibility-with-vitest-5 branch from f511dee to c3a356b Compare July 1, 2026 12:57

greptile-apps Bot reviewed Jul 19, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support vitest 5#86

Support vitest 5#86
GuillaumeLagrange wants to merge 4 commits into
mainfrom
cod-2931-prepare-compatibility-with-vitest-5

GuillaumeLagrange commented Jun 29, 2026

Uh oh!

sheremet-va Jun 30, 2026

Uh oh!

GuillaumeLagrange Jul 1, 2026

Uh oh!

GuillaumeLagrange Jul 1, 2026

Uh oh!

sheremet-va Jul 1, 2026

Uh oh!

GuillaumeLagrange Jul 1, 2026

Uh oh!

codspeed-hq Bot commented Jul 1, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 19, 2026 •

edited

Loading

Flowchart

Comments Outside Diff (2)

Uh oh!

greptile-apps Bot Jul 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	// TODO: Check if this can be avoided
	// TODO(COD-XXX): Check if this can be avoided

Uh oh!

Conversation

GuillaumeLagrange commented Jun 29, 2026

Uh oh!

sheremet-va Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

GuillaumeLagrange Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

GuillaumeLagrange Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

sheremet-va Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

GuillaumeLagrange Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

codspeed-hq Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will regress 2 benchmarks

Performance Changes

Footnotes

Uh oh!

greptile-apps Bot commented Jul 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Important Files Changed

Flowchart

Comments Outside Diff (2)

Uh oh!

greptile-apps Bot Jul 19, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codspeed-hq Bot commented Jul 1, 2026 •

edited

Loading

greptile-apps Bot commented Jul 19, 2026 •

edited

Loading