feat(test-runner): support named test locks by yury-s · Pull Request #41862 · microsoft/playwright

yury-s · 2026-07-18T00:05:02Z

Adds lock: string | string[] to test details, also available on test.describe groups.

Tests that share a lock name never run concurrently, while all other tests continue to run in parallel. Locks work across files and projects.
All locks of a test group are acquired before the job starts and released when it finishes, so locks cannot deadlock and waiting for a lock does not consume the test timeout.
Test grouping and sharding are not affected by locks. In default and serial modes the lock is held for the duration of the whole file group.

Skn0tt

Nice idea with using locks in the group key!

It's hard to imagine that differentiating between the empty Set and undefined measurably helps. If that was unintended, maybe we can simplify it away.

I'll look at declaring locks on fixtures on top of this.

Tests that share a lock name declared in test details never run concurrently, while all other tests continue to run in parallel. Locks work across files and projects. All locks of a test group are acquired atomically when the job is scheduled and released when it finishes, so locks cannot deadlock, and waiting for a lock does not consume the test timeout. Fixes: microsoft#21484

Fold lock inheritance into the existing bindFileSuiteToProject pass instead of a separate lazy ancestor walk, group tests with an identical lock set into a single job so beforeAll/afterAll hooks run once, and avoid allocations on the scheduling path when no locks are declared.

… case

yury-s · 2026-07-22T18:05:26Z

It's hard to imagine that differentiating between the empty Set and undefined measurably helps. If that was unintended, maybe we can simplify it away.

There is no good reason, simplified.

dgozman · 2026-07-22T19:18:01Z

    // There are 3 kinds of parallel tests:
    // - Tests belonging to parallel suites, without beforeAll/afterAll hooks.
    //   These can be run independently, they are put into their own group, key === test.
+    // - Tests with locks are grouped by their lock signature instead, key === signature,


This rearranges tests based on locks. I am afraid that can have unintended consequences. For example, adding/removing a lock will affect which tests get into which shard. With dynamic locks based on env or something, we might run into problems.

Are we sure this improves scheduling (performance?) in any meaningful way? Why do we think grouping by locks is more beneficial than avoiding the same beforeAll hook running multiple times?

As discussed offline, reverted changes to the grouping based on lock presence. If it results in an inefficiency in the wild, we'll address it later.

Do not affect test grouping based on locks, drop redundant comments, trim the docs.

github-actions · 2026-07-22T20:50:43Z

Test results for "tests 1"

6 flaky

⚠️ [chromium-library] › library/video.spec.ts:736 › screencast › should work with video+trace `@chromium-ubuntu-22.04-arm-node20`
⚠️ [chromium-library] › library/video.spec.ts:699 › screencast › should capture full viewport on hidpi `@realtime-time-library-chromium-linux`
⚠️ [chromium-library] › library/chromium/chromium.spec.ts:211 › should intercept service worker requests (main and within) `@chromium-ubuntu-22.04-node24`
⚠️ [chromium-library] › library/chromium/chromium.spec.ts:211 › should intercept service worker requests (main and within) `@chromium-ubuntu-22.04-node20`
⚠️ [chromium-library] › library/video.spec.ts:356 › screencast › should work for popups `@chromium-ubuntu-22.04-node22`
⚠️ [playwright-test] › ui-mode-trace.spec.ts:827 › should update state on subsequent run `@macos-latest-node22`

50249 passed, 1190 skipped

Merge workflow run.

github-actions · 2026-07-22T20:53:52Z

Test results for "MCP"

2 failed
❌ [firefox] › mcp/cli-core.spec.ts:189 › dialog-accept @mcp-windows-latest-firefox
❌ [firefox] › mcp/dashboard.spec.ts:108 › should activate session when show is called with -s @mcp-windows-latest-firefox

7784 passed, 1262 skipped

Merge workflow run.

github-actions · 2026-07-22T21:00:32Z

Hi, I'm the Playwright bot and I took a look at the CI failures.

🟡 Two Firefox/Windows MCP failures — one is a proven flake, the other I can't fully clear (no path from this diff either way)

The latest MCP report shows 2 failed, both on mcp-windows-latest-firefox in the same shard: mcp/dashboard.spec.ts:108 › should activate session when show is called with -s and mcp/cli-core.spec.ts:189 › dialog-accept <prompt>. The dashboard one also fails on main, so it's a pre-existing flake. The other only shows up on this branch, so I can't call it confirmed — but this PR is a test-runner locks change with no path into MCP behavior. Leaning clear; a rerun of that shard would settle it.

Details

Overall: This PR (feat(test-runner): support named test locks) touches common/, runner/dispatcher.ts, runner/testGroups.ts, docs, types, and a new tests/playwright-test/test-locks.spec.ts. It touches no MCP code. The active lock path only fires for tests that declare a lock; MCP tests declare none, and the _scheduleJobs refactor is behaviorally equivalent for lock-free jobs. Both failures are Firefox-on-Windows only, in one shard.

Pre-existing flake / infra

[firefox] › mcp/dashboard.spec.ts:108 › should activate session when show is called with -s @mcp-windows-latest-firefox — flake. Same test, same error (toHaveAccessibleName … element(s) not found, 5s timeout) failed on main at head 244a1ffd (run 29945242893, pr_number null) as well as here at head e1418e14 (run 29953886735). A failure on main can't be caused by this branch, so this one is confirmed pre-existing.

Uncertain

[firefox] › mcp/cli-core.spec.ts:189 › dialog-accept <prompt> @mcp-windows-latest-firefox — failed with expect(received).toContain(expected) … Received has value: undefined. In the results DB this test is otherwise green (Firefox: failed 1 of 399 runs, and that one failure is this PR's run), so I can't cite an independent occurrence and won't call it a confirmed flake under our rule. But it's not something the diff reaches: it declares no lock, exercises no scheduling, and lives in MCP code the PR doesn't touch. It failed in the same Firefox/Windows shard as the proven-flaky dashboard test — a bad-shard signature, not a locks regression (which would stall many tests, not miss one dialog assertion). What would make me sure: a green rerun of the mcp-windows-latest-firefox shard on this head.

The tests 1 reports over this run were flaky-only (video screencast, oopif, service-worker, ui-mode-trace) with 0 real failures — all rescued on retry and unrelated to test scheduling.

_{Triaged by the Playwright bot.}

_{Triaged by the Playwright bot - agent run}

This comment has been minimized.

Sign in to view

yury-s requested review from Skn0tt and dgozman July 20, 2026 16:32

github-actions Bot mentioned this pull request Jul 21, 2026

fix(screencast): acknowledge frames after callbacks #41898

Merged

Skn0tt approved these changes Jul 22, 2026

View reviewed changes

yury-s added 4 commits July 22, 2026 10:33

improvements

62e1715

chore(test-runner): always allocate lock sets, drop undefined special…

af276a0

… case

yury-s force-pushed the fix-21484 branch from 89ed0af to af276a0 Compare July 22, 2026 17:40

chore(test-runner): shorten comments

5e230e3

This comment has been minimized.

Sign in to view

dgozman approved these changes Jul 22, 2026

View reviewed changes

chore(test-runner): address review comments

2fff368

Do not affect test grouping based on locks, drop redundant comments, trim the docs.

github-actions Bot mentioned this pull request Jul 23, 2026

fix: correct pending navigation log spacing #41949

Merged

Uh oh!

Conversation

yury-s commented Jul 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Skn0tt left a comment

Choose a reason for hiding this comment

Uh oh!

yury-s commented Jul 22, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dgozman Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

yury-s Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 22, 2026

Test results for "tests 1"

Uh oh!

github-actions Bot commented Jul 22, 2026

Test results for "MCP"

Uh oh!

github-actions Bot commented Jul 22, 2026

🟡 Two Firefox/Windows MCP failures — one is a proven flake, the other I can't fully clear (no path from this diff either way)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yury-s commented Jul 18, 2026 •

edited

Loading