Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
250 changes: 250 additions & 0 deletions proposals/proposal - unbreak test cli arg.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,250 @@
# Proposal: un/break `--test`

Currently, this flag's behaviour is very unexpected, often resulting in silent-failure footguns, and precludes command nesting, as demonstrated in a very simple and typical CI setup:

```yaml title="unit & e2e tests"
- name: tests with coverage
run: node --run test -- --test-reporter lcov
```

That results in

```sh
node --test --test-reporter lcov
```

In most cases, `--test-reporter` gets lost (the `lcov` reporter is not enabled); in a worse case, this unexpectedly causes tests within a directory named `lcov` to be run.

## More info

Currently (since always):

`--test` receives everything after it (space-delimited).

`--test` currently does 2 things:

* enables the test runner
* accepts paths for the runner to consume

This is similar to another existing feature with which we want to improve interop: `watch` mode.

`--watch` currently does 2 things:

* enables `watch` mode
* optionally accepts 1 value to override the entrypoint

```sh
node
--watch
--watch-path ./src/**/*.ts
--watch-path ./test/**/*.ts
```

## Proposed options

### Option: Break all the things

Currently `watch` has 2 flags (which are redundant):

```sh title='reads "main" etc from package.json'
node --watch
```
```sh
node --watch ./src/not-pjson-main.js
```
```sh
node
--watch ./src/not-pjson-main.js
--watch-path ./src/**/*.js
```

This could be simplified into just 1 `--watch` flag (where the default value is a single-element array of the derived entrypoint).

```sh title='reads "main" etc from package.json'
node --watch
```
```sh title='override derived entrypoint'
node --watch ./src/not-pjson-main.js
```
```sh title='package.json "main" + an additional path'
node --watch ./src/**/*.js
```
```sh title='explicit entrypoint + additional paths'
node
--watch ./assets/**
--watch ./src/**/*.js
./src/not-pjson-main.js
```

An explicit entrypoint, like all node commands, must come last and must be a relative or absolute path (not a glob/pattern).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this tracks for test. Many users will only want to pass a single glob pattern; at least I know that I do! What would the explicit entry point do for a test script?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, okay, fair. So when watch mode is combined with test mode, shall we say that entry-point is not applicable—it's only applicable for watch mode separately?


`test` would then work the same way:

```sh title='reads "main" etc from package.json'
node --test
```
```sh title='package.json "main" + an additional path'
node --test ./src/**/*.test.js
```
```sh title='explicit entrypoint + an additional path'
node
--test ./src/foo/*.test.js
--test ./src/bar/*.test.js
./src/not-pjson-main.js

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would the purpose of this be? The test runner runs for the two glob patterns and then does what with this entry point script? Executes it as a test too or just as a regular node script?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes, true. Let's say there is no such thing as an entry-point for test mode (I think that doesn't exist currently either). If there's any "entry-point" when --test is present, throw.

I think in order to run an arbitrary script, it should be done with --import (and friends); otherwise, you get into inception.

node
  --import ./test/env.setup.js
  --import ./test/fix-snapshot-naming.js
  --test ./src/foo/*.test.js
  --test ./src/bar/*.test.js

```

#### All together

`--watch` receives a clone of `--test`'s resolved value(s) **plus** `--watch`'s own values—with 1 exception: when both `test` and `watch` modes are enabled, `--watch` does not include a default entrypoint (it's irrelevant and it would likely result in perf waste at best, and unexpected behaviour at worst).

All examples are sequence-independent (all within the same heading behave the same).

##### Watch + test paths & main entrypoint:

```sh
node
--test ./src/foo/*.test.js
--test ./src/bar/*.test.js
--watch
./src/not-pjson-main.js
```
```sh
node
--watch
--test ./src/foo/*.test.js
--test ./src/bar/*.test.js
./src/not-pjson-main.js
```

##### Without an entrypoint (just paths):

```sh
node
--test ./src/foo/*.test.js
--test ./src/bar/*.test.js
--watch
```
```sh
node
--watch
--test ./src/foo/*.test.js
--test ./src/bar/*.test.js
```

##### _Additional_ watch paths

```sh
node
--test ./src/foo/*.test.js
--test ./src/bar/*.test.js
--watch ./src/**/*.js
```
```sh
node
--watch ./src/**/*.js
--test ./src/foo/*.test.js
--test ./src/bar/*.test.js
```

### Option: Apply `--watch`'s current design to `--test`

`--test` is optional and optionally accepts 1 value, whose value defaults to node's built-in glob defaults (`./src/**/*.test.(c|m)?(j|t)s`, etc). Additional paths are supplied via a new `--test-path` flag:

<table>
<thead>
<tr>
<th>Description</th>
<th>Code sample</th>
<th>Resulting <code>--test</code> value</th>
</tr>
</thead>

<tbody>

<tr>
<td>
Override default
</td>
<td>

```sh
node --test ./src/foo/*.test.js
```

</td>
<td>

`./src/foo/*.test.js`

</td>
</tr>

<tr>
<td>
Override default & Set additional path
</td>
<td>

```sh
node
--test ./src/foo/*.test.js
--test-path ./src/bar/*.test.js
```

</td>
<td>

`./src/foo/*.test.js`
`./src/bar/*.test.js`

</td>
</tr>

<tr>
<td>
Defaults + Additional paths
</td>
<td>

```sh
node
--test
--test-path ./src/foo/*.test.js
--test-path ./src/bar/*.test.js
```

</td>
<td>

`./src/**/*.test.(c|m)?(j|t)s`
~`./src/foo/*.test.js`~
~`./src/bar/*.test.js`~

which reduces to only the default: `./src/**/*.test.(c|m)?(j|t)s`

</td>
</tr>

<tr>
<td>
No defaults & Additional paths
</td>
<td>

```sh
node
--test-path ./src/foo/*.test.js
--test-path ./src/bar/*.test.js
```

</td>
<td>

`./src/foo/*.test.js`
`./src/bar/*.test.js`

</td>
</tr>

</tbody>
</table>