doc(proposal): add expectFailure enhancements #10

Han5991 · 2026-01-29T02:23:59Z

This PR adds a proposal for enhancing expectFailure to support both custom messages (for reasoning) and validation (matching errors via regex/objects), addressing the needs discussed in nodejs/node#61563.

vassudanagunta · 2026-01-29T03:33:30Z

@Han5991 this should reference nodejs/node#61570 which goes beyond nodejs/node#61563. It may end up being the case that a fix for the latter lands before the former, since the latter is relatively trivial and simply brings expectFailure in line with todo/skip.

--EDIT--

I'm a little confused. Your first proposal, #9, included a way to supply both a reason and an expected error as I proposed in nodejs/node#61570, but this one does not.

…1570 requirement)

Han5991 · 2026-01-29T04:41:52Z

@vassudanagunta

Sorry for the confusion. PR #9 was accidentally opened from my other account, so I closed it and opened this one instead.
I have updated this proposal to include the Object support (both reason and validation) as you suggested in #61570. Thanks for pointing that out!

vassudanagunta · 2026-01-29T13:03:31Z

Alternative syntaxes to consider for supplying both reason and expected error:

test('fails with reason and specific error', {
  expectFailure: 'Bug #123: Edge case behavior',
  expectFailureMessage: /err msg pattern/
}, () => ...);

or for consistency with assert.throws:

test('fails with reason and specific error', {
  expectFailure: 'Bug #123: Edge case behavior',
  expectFailureError: <RegExp> | <Function> | <Object> | <Error>
}, () => ...);

The user can supply either one of the two new options or both.

vassudanagunta · 2026-01-29T13:17:02Z

Consistency with assert.throws could be trivial to implement either by sharing code or by calling assert.throws in a try-catch to invert the effect.

Han5991 · 2026-01-29T14:38:28Z

Thanks for the feedback!

Implementation:
Great suggestion! I plan to update the implementation to use assert.throws internally for validation. This will allow us to leverage all the existing validation capabilities (RegExp, Object, Class, custom function) without reinventing the wheel and keep the code cleaner.
API Syntax:
Regarding the flat options (expectFailureError), I lean towards keeping the nested object structure { expectFailure: { with: ..., message: ... } }. It keeps all failure expectations grouped under a single property and seems more extensible if we need to add more configuration in the future, without polluting the top-level options namespace.

vassudanagunta · 2026-01-29T14:42:51Z

the terseness and legibility of with+message is very nice. as i said on your PR, it gets my top vote.

Update the `expectFailure` enhancements proposal based on feedback and implementation alignment: - Consolidate validation logic under the `with` property within an object. - Remove direct RegExp support in favor of the object syntax for consistency. - Specify usage of `assert.throws` for robust error validation. - Document alternatives considered (flat options).

JakobJingleheimer · 2026-01-29T16:18:42Z

IIR, we had originally envisioned expectFailure would eventually accept a string for reason, and an object or regex for a matcher:

it('should do the thing', { expectFailure: 'chromium#1234' }, () => {…});

it('should do another thing', { expectFailure: /some telltale needle/ }, () => {…});

it('should do something else', { expectFailure: new RangeError(…) }, () => {…});

should do the thing # EXPECT FAILURE "chromium#1234"

should do another thing # EXPECT FAILURE matching /some telltale needle/

should do something else # EXPECT FAILURE matching RangeError(…)

Where matcher would follow the behaviour of assert.throws (and would probably leverage it under the hood).

proposals/expect-failure-enhancements.md

Han5991 · 2026-01-30T02:22:31Z

IIR, we had originally envisioned expectFailure would eventually accept a string for reason, and an object or regex for a matcher:
it('should do the thing', { expectFailure: 'chromium#1234' }, () => {…});

it('should do another thing', { expectFailure: /some telltale needle/ }, () => {…});

it('should do something else', { expectFailure: new RangeError(…) }, () => {…});
should do the thing # EXPECT FAILURE "chromium#1234"

should do another thing # EXPECT FAILURE matching /some telltale needle/

should do something else # EXPECT FAILURE matching RangeError(…)
Where matcher would follow the behaviour of assert.throws (and would probably leverage it under the hood).

@JakobJingleheimer

I've updated the proposal to reflect the latest feedback. Ready for re-review!

proposals/expect-failure-enhancements.md

Explicitly state that passing an empty object ({}) to expectFailure should throw ERR_INVALID_ARG_VALUE to prevent ambiguity and potential user error.

vassudanagunta · 2026-01-31T08:29:01Z

RE the back and forth on whether expectFailure should be truthy, I think it should be an open question to discuss, either with input from other voices here in comments or in the test runner team meeting. Here is my opinion copied from the PR:

I agree with @ljharb in general, but in this specific case it will result in assertFailure behaving differently from todo and skip (see the resolved comment). I think that will be odd/confusing.

Also, given the specificity of the linked todo/skip logic, that behavior seems rather intentional. 🤷🏾‍♂️

JakobJingleheimer

Thanks for this!

I think some whitespace is amiss (e.g. too many spaces between * and the text, too much indentation, missing blank line between heading and its content, etc).

proposals/expect-failure-enhancements.md

JakobJingleheimer · 2026-02-02T20:54:27Z

proposals/expect-failure-enhancements.md

+    message: 'Bug #123: Edge case behavior', // Reason
+    with: /Index out of bounds/              // Validation


I think these keys are a bit ambiguous. I think message → label and with → match (or matcher or matching, or perhaps error?)

proposals/expect-failure-enhancements.md

JakobJingleheimer · 2026-02-02T21:01:54Z

proposals/expect-failure-enhancements.md

+          'must not be an empty object'
+        );
+        ```
+    *   If the object contains `with` or `message` properties → **Configuration Object**.


I think if it contains either or both but no more → config object

proposals/expect-failure-enhancements.md

- Update property names to match/label in description to match code examples. - Fix markdown formatting (whitespace, blank lines) as requested.

Han5991 · 2026-02-02T22:51:53Z

Thanks for the detailed review! I've updated the proposal to address your feedback

JakobJingleheimer

🙌 LGTM aside from the small revert needed because of my mistaken assumption.

JakobJingleheimer · 2026-02-02T22:56:13Z

proposals/expect-failure-enhancements.md

-When a **RegExp**, **Class** (Function), or **Error Object** is provided directly, it acts as the validation logic. This leverages `assert.throws` behavior directly.
+### Matcher: Error constructor, Error-like object, RegExp, or validation function
+
+When a `Error` constructor, `Error`-like object, `RegExp`, or validation function is provided, it is treated as validation to match against, mirroring the behaviour of [`assert.throws`](https://nodejs.org/docs/latest-v25.x/api/assert.html#assertthrowsfn-error-message) (possibly leveraging it under the hood).


nit 😅

Suggested change

When a `Error` constructor, `Error`-like object, `RegExp`, or validation function is provided, it is treated as validation to match against, mirroring the behaviour of [`assert.throws`](https://nodejs.org/docs/latest-v25.x/api/assert.html#assertthrowsfn-error-message) (possibly leveraging it under the hood).

When an `Error` constructor, `Error`-like object, `RegExp`, or validation function is provided, it is treated as validation to match against, mirroring the behaviour of [`assert.throws`](https://nodejs.org/docs/latest-v25.x/api/assert.html#assertthrowsfn-error-message) (possibly leveraging it under the hood).

JakobJingleheimer · 2026-02-02T22:58:52Z

proposals/expect-failure-enhancements.md

-*   The feature is **disabled** only if `expectFailure` is `undefined` or `false`.
-*   **All other values** enable the feature (treat as truthy).
-    *   `expectFailure: ''` (Empty String) → **Enabled** (treats as generic failure expectation).
-    *   `expectFailure: 0` → **Enabled** (treated as a Matcher Object unless specific logic excludes numbers, but per consistency it enables the feature).
+* The feature is **disabled** if `expectFailure` is **falsy** (e.g., `false`, `null`, `undefined`, `0`, `''`).
+* **Truthy values** enable the feature.


Whoops, sorry, I was wrong about this: #10 (comment)

You can revert this part of the change—it was good/correct before.

ljharb

LGTM overall

ljharb · 2026-02-02T23:25:55Z

proposals/expect-failure-enhancements.md

+```
+- **Behavior**: The test is expected to fail. The string is treated as a label/reason.
+- **Validation**: None. It accepts _any_ error.
+- **Output**: The reporter displays the string literally (e.g., `# EXPECTED FAILURE Bug #123…`).


Suggested change

- **Output**: The reporter displays the string literally (e.g., `# EXPECTED FAILURE Bug #123…`).

- **Output**: The reporter displays the string literally, prefixed by a TAP comment character (e.g., `# EXPECTED FAILURE Bug #123…`).

ljharb · 2026-02-02T23:27:43Z

proposals/expect-failure-enhancements.md

+- **Properties**:
+    - `label` (String): The failure reason/label (displayed in reporter).
+    - `match` (RegExp | Object | Function | Class): Validation logic. This is passed directly to `assert.throws` validation argument, supporting all its capabilities.
+- **Behavior**: The test passes **only if** the error matches the `match` criteria.
+- **Output**: The reporter displays the `label`.


I think that either one of these properties must be present, or it should throw - iow, using the object form with just a label is fine, because it reduces diff churn if match is added or removed later; and using the object form with just match is fine, because it reduces diff churn if label is added later - but if you have neither, it's far more likely to be a bug than intentional behavior.

(update: i see further down that this is the case, but imo it should also mention here that one of these two properties must be present)

feat: add expectFailure enhancements proposal

036ee9c

Han5991 mentioned this pull request Jan 29, 2026

test_runner: support custom message for expectFailure nodejs/node#61563

Closed

docs: add object support for combined reason and validation (fixes #6…

049c2d3

…1570 requirement)

Han5991 force-pushed the proposals/expect-failure-reason branch from 0de23a6 to abb5912 Compare January 29, 2026 21:55

vassudanagunta suggested changes Jan 29, 2026

View reviewed changes

proposals/expect-failure-enhancements.md Outdated Show resolved Hide resolved

docs: update expectFailure proposal with direct matchers and rationale

87802e3

Han5991 force-pushed the proposals/expect-failure-reason branch from 6e1e4a8 to 87802e3 Compare January 30, 2026 02:21

docs: update expectFailure empty string behavior for consistency

cb12492

Han5991 requested a review from vassudanagunta January 30, 2026 02:35

vassudanagunta suggested changes Jan 30, 2026

View reviewed changes

proposals/expect-failure-enhancements.md Show resolved Hide resolved

proposals/expect-failure-enhancements.md Outdated Show resolved Hide resolved

docs: expand equivalence examples and clarify activation logic

550464a

Han5991 requested a review from vassudanagunta January 30, 2026 04:21

vassudanagunta approved these changes Jan 30, 2026

View reviewed changes

JakobJingleheimer changed the title ~~feat: add expectFailure enhancements proposal~~ doc(proposal): add expectFailure enhancements Jan 30, 2026

doc: disallow empty object in expectFailure option

38976f0

Explicitly state that passing an empty object ({}) to expectFailure should throw ERR_INVALID_ARG_VALUE to prevent ambiguity and potential user error.

JakobJingleheimer reviewed Feb 2, 2026

View reviewed changes

doc: apply review feedback to expectFailure proposal

4e2c487

- Update property names to match/label in description to match code examples. - Fix markdown formatting (whitespace, blank lines) as requested.

Han5991 requested a review from JakobJingleheimer February 2, 2026 22:50

JakobJingleheimer reviewed Feb 2, 2026

View reviewed changes

JakobJingleheimer requested review from a team, ljharb and pmarchini February 2, 2026 23:00

ljharb reviewed Feb 2, 2026

View reviewed changes

		message: 'Bug #123: Edge case behavior', // Reason
		with: /Index out of bounds/ // Validation

	When a `Error` constructor, `Error`-like object, `RegExp`, or validation function is provided, it is treated as validation to match against, mirroring the behaviour of [`assert.throws`](https://nodejs.org/docs/latest-v25.x/api/assert.html#assertthrowsfn-error-message) (possibly leveraging it under the hood).
	When an `Error` constructor, `Error`-like object, `RegExp`, or validation function is provided, it is treated as validation to match against, mirroring the behaviour of [`assert.throws`](https://nodejs.org/docs/latest-v25.x/api/assert.html#assertthrowsfn-error-message) (possibly leveraging it under the hood).

	- Output: The reporter displays the string literally (e.g., `# EXPECTED FAILURE Bug #123…`).
	- Output: The reporter displays the string literally, prefixed by a TAP comment character (e.g., `# EXPECTED FAILURE Bug #123…`).

Uh oh!

doc(proposal): add expectFailure enhancements #10

Are you sure you want to change the base?

doc(proposal): add expectFailure enhancements #10

Conversation

Han5991 commented Jan 29, 2026

Uh oh!

vassudanagunta commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Han5991 commented Jan 29, 2026

Uh oh!

vassudanagunta commented Jan 29, 2026

Uh oh!

vassudanagunta commented Jan 29, 2026

Uh oh!

Han5991 commented Jan 29, 2026

Uh oh!

vassudanagunta commented Jan 29, 2026

Uh oh!

JakobJingleheimer commented Jan 29, 2026

Uh oh!

Uh oh!

Han5991 commented Jan 30, 2026

Uh oh!

Uh oh!

Uh oh!

vassudanagunta commented Jan 31, 2026

Uh oh!

JakobJingleheimer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JakobJingleheimer Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JakobJingleheimer Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Han5991 commented Feb 2, 2026

Uh oh!

JakobJingleheimer left a comment

Choose a reason for hiding this comment

Uh oh!

JakobJingleheimer Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

JakobJingleheimer Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

ljharb left a comment

Choose a reason for hiding this comment

Uh oh!

ljharb Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

ljharb Feb 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vassudanagunta commented Jan 29, 2026 •

edited

Loading