Skip to content

fix(test): increase Intel macOS timeout#41851

Merged
Skn0tt merged 2 commits into
microsoft:mainfrom
Skn0tt:skn0tt-adjust-test-timeouts
Jul 20, 2026
Merged

fix(test): increase Intel macOS timeout#41851
Skn0tt merged 2 commits into
microsoft:mainfrom
Skn0tt:skn0tt-adjust-test-timeouts

Conversation

@Skn0tt

@Skn0tt Skn0tt commented Jul 17, 2026

Copy link
Copy Markdown
Member

This is based on 285 main push runs from the test-results database. Intel macOS runners regularly push unrelated library tests close to the 30s timeout, so this raises their default to 60s.

It also removes six .slow() calls that are no longer needed with the new default, or whose tests consistently finish well below 30s.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 696f2d8a-c16c-4d90-8353-8e33d66a2c31
@Skn0tt
Skn0tt requested a review from dgozman July 17, 2026 15:46
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

annotation: { type: 'issue', description: 'https://github.com/microsoft/playwright/issues/41216' },
}, async ({ browserType, createUserDataDir, browserName }) => {
it.fixme(browserName === 'firefox');
it.slow();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This effectively decreases timeout from 90s to 60s, maybe leave it as is?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! Turns out this was added while #41224 was causing Firefox to hang. #41309 interpreted that as two slow launches, but two days later we reverted the change and marked this test as fixme on Firefox. The .slow() stayed behind.

Of 3,338 recent executions, only one exceeded 30s: it took 79s during a failed run with 154 other timeouts. Excluding that broken run, p95 is 3.7s and the max is 17.4s. I think removing it is right - the original reason no longer applies.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 696f2d8a-c16c-4d90-8353-8e33d66a2c31
@Skn0tt
Skn0tt merged commit e23ccf3 into microsoft:main Jul 20, 2026
44 of 46 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

Test results for "MCP"

2 failed
❌ [chrome] › mcp/annotate.spec.ts:173 › user-initiated annotate downloads zip with feedback.md @mcp-macos-latest-chrome
❌ [chrome] › mcp/annotate.spec.ts:230 › should capture annotations via show --annotate @mcp-macos-latest-chrome

7758 passed, 1249 skipped


Merge workflow run.

@github-actions

Copy link
Copy Markdown
Contributor

Test results for "tests 1"

1 failed
❌ [playwright-test] › reporter-html.spec.ts:3402 › created › speedboard › clicking on label should not exit speedboard @macos-latest-node22

4 flaky ⚠️ [chromium-library] › library/beforeunload.spec.ts:130 › should support dismissing the dialog multiple times `@frozen-time-library-chromium-linux`
⚠️ [chromium-library] › library/video.spec.ts:495 › screencast › should capture static page in persistent context @smoke `@chromium-ubuntu-22.04-node22`
⚠️ [firefox-page] › page/page-emulate-media.spec.ts:144 › should keep reduced motion and color emulation after reload `@firefox-ubuntu-22.04-node20`
⚠️ [playwright-test] › ui-mode-trace.spec.ts:778 › should partition action tree state by test `@windows-latest-node22`

49958 passed, 1186 skipped


Merge workflow run.

@github-actions

Copy link
Copy Markdown
Contributor

Hi, I'm the Playwright bot and I took a look at the CI failures.

🟢 Both failures are pre-existing flakes — this PR is clear

The MCP run failed two mcp/annotate.spec.ts tests on mcp-macos-latest-chrome. Both flake across unrelated main push runs and other PRs, so this PR isn't the cause. The one caveat worth calling out — this PR removes test.slow() from the failing :230 test — doesn't hold up: the failure isn't a timeout.

Details

Pre-existing flake / infra

  • [chrome] › mcp/annotate.spec.ts:230 › should capture annotations via show --annotate — the PR does remove test.slow() from this exact test, so it's the natural suspect. But the failure signature is Error: Cannot read properties of undefined (reading 'endpoint') — a race, not a timeout. In the test-results DB, 44 of 45 recorded failures hit this same error with the old 90000ms slow budget still in place, so the shorter default isn't what breaks it. It flakes everywhere: mcp-macos-latest-firefox 14/288, -webkit 9/293, -chromium 6/293, -chrome 6/293 runs, and on ubuntu too — across 17 distinct main push SHAs (no PR) plus 23 other PRs.

  • [chrome] › mcp/annotate.spec.ts:173 › user-initiated annotate downloads zip with feedback.md — this PR doesn't touch it. Flaky across the DB: mcp-macos-latest-firefox 20/288, -webkit 15/293, -chrome 8/293 runs, over 17 main push SHAs and 27 other PRs.

Assessment: the diff is confined to timeout/test.slow() adjustments. The library-suite timeout bump (tests/library/playwright.config.ts) doesn't reach the MCP suite, which has its own config. The only overlap is the removed test.slow() on :230 — and since that failure is a non-timeout race that reproduces under the longer budget, the removal isn't responsible. Both tests have a clear cross-SHA flake history on main.

Triaged by the Playwright bot - agent run

@github-actions

Copy link
Copy Markdown
Contributor

Hi, I'm the Playwright bot and I took a look at the CI failures.

🟡 Two MCP failures are known flakes; one HTML-reporter failure I can't clear

The two MCP failures are well-documented pre-existing flakes. The reporter-html speedboard failure is in the playwright-test suite, which this timeout PR doesn't reach — but it has a spotless history, so I can't positively prove it's a flake. Calling it uncertain.

Details

This PR only touches tests/library/playwright.config.ts (raises the Intel-macOS timeout) and drops a few test.slow() markers in the library and MCP suites. It changes no product code and no playwright-test config.

Pre-existing flake / infra

  • [chrome] › mcp/annotate.spec.ts:230 › should capture annotations via show --annotate — pre-existing flake. It fails on 30+ unrelated PRs and push SHAs in the test-results DB (~1% per browser), always with the same root error Cannot read properties of undefined (reading 'endpoint') that then hangs to the test timeout. This PR does remove this test's test.slow(), which shrinks its budget from 90s to 30s — but the historical failures time out at 90000ms too, so the hang fails regardless of the budget. Not caused by the PR.
  • [chrome] › mcp/annotate.spec.ts:173 › user-initiated annotate downloads zip with feedback.md — pre-existing flake, untouched by this PR. Fails across browsers on unrelated SHAs (firefox 5, webkit 4, chromium 2, chrome 1 of ~290–300 runs each).

Uncertain

  • [playwright-test] › reporter-html.spec.ts:3402 › created › speedboard › clicking on label should not exit speedboard (macos-latest-node22) — failed once with an aria-snapshot mismatch and self-rescued on rerun on the same bot. The diff can't mechanistically reach it: it lives in the playwright-test suite with its own 75s timeout, while this PR's changes are scoped to the library config and library/MCP test.slow() markers. But its history is 409 runs, 0 prior failures — I can't cite the same test failing anywhere the PR isn't responsible, so I won't call it a confirmed flake. To be sure, a clean rerun of tests 1 would settle it.

Assessment: nothing here points at a real regression from the timeout change. The two MCP failures are established cross-PR flakes, and the lone playwright-test failure sits outside the code this PR touches and already passed on retry — I just can't prove it pre-existing from the data.

Triaged by the Playwright bot - agent run

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants