test: opt the test suite out of telemetry - #1668
Conversation
34 test files spawn the real CLI across ~62 call sites. Each spawn runs the preAction hook exactly like a user invocation, so a local `pnpm test` persisted an anonymousId into the developer's real global config (~/.config/openspec/config.json) and POSTed a command_executed event per spawn to the telemetry endpoint. CI never saw this because CI=<truthy> already disables telemetry; it only happens on contributor machines, where it also skews the maintainers' usage data with test traffic. Set OPENSPEC_TELEMETRY=0 / DO_NOT_TRACK=1 via vitest's env so workers and the CLI children they spawn are both covered. Telemetry's own tests delete these vars before asserting, so they are unaffected.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughVitest now sets ChangesTest telemetry isolation
Estimated code review effort: 1 (Trivial) | ~2 minutes Merge Risk: ⚪ Minimal · up to The change limits telemetry side effects to test runs without altering normal user invocations, and the reported test and lint checks pass. No actionable merge-blocking risk remains beyond normal review. Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Problem
34 test files spawn the real CLI across ~62 call sites. Each spawn runs the
preActionhook exactly like a user invocation, so runningpnpm testlocally:~/.config/openspec/config.jsongains a persistedanonymousIdandnoticeSeencommand_executedevent per spawn toedge.openspec.devanonymousIdis only ever written bygetOrCreateAnonymousId(), which is called solely fromtrackCommand()after theisTelemetryEnabled()gate — so its presence is proof that events were sent, not just that a file was touched.CI never sees this.
isCiEnvironment()disables telemetry wheneverCIis set, so this only happens on contributor machines. The side effects are that contributors emit analytics they didn't opt into by running the test suite, and the maintainers' usage data gets skewed by hundreds of test-generated events per run.Reproduction
Against
main(2826b88), with an isolatedHOMEso the write is visible and doesn't touch the real config:One test file is enough to trigger it.
Fix
Set
OPENSPEC_TELEMETRY=0/DO_NOT_TRACK=1through vitest'stest.env. Workers inherit it, and so do the CLI child processes they spawn — verified below rather than assumed. Telemetry's own tests callenableTelemetry(), which deletes these vars before asserting, so they keep their coverage.Verification
mainHOMEanonymousIdpersistedpnpm testpnpm test— 136/136 files, 3969/3969 tests passpnpm lint— cleanHOMEwrites noconfig.jsonanywheretest/telemetry/*still passes (it manages its own env and mocksfetch)Note on the interaction with #1666
On
main, that same isolated-HOMErun also failsspec show > should display spec in text format— the stdout-pollution bug in #1666. This PR silences that failure as a side effect, because a disabled-telemetry run never prints the notice at all.That is a masking effect, not a fix: #1666 is the user-facing bug (real users have telemetry enabled and no
CIset), and it needs to land on its own. I'd suggest treating these as independent — this one for test hygiene, #1666 for the actual stdout contract. #1666 also adds a unit assertion pinning the notice to stderr, so the behavior stays covered even with telemetry off in the suite.Happy to rebase this on top of #1666, or hold it until that one lands, whichever you prefer.
Summary by CodeRabbit