Skip to content

Commit c3d28e3

Browse files
chariskcharisk
committed
Move testing docs to separate file
1 parent 840cfbf commit c3d28e3

File tree

2 files changed

+132
-129
lines changed

2 files changed

+132
-129
lines changed

‎CONTRIBUTING.md‎

Lines changed: 1 addition & 129 deletions
Original file line numberDiff line numberDiff line change
@@ -93,135 +93,7 @@ More information about Storybook can be found inside the **Overview** page once
9393

9494
### Testing
9595

96-
We have several types of tests:
97-
98-
* Unit tests: these live in the `tests/unit-tests/` directory
99-
* View tests: these live in `src/view/variant-analysis/__tests__/`
100-
* VSCode integration tests:
101-
* `test/vscode-tests/activated-extension` tests: These are intended to cover functionality that require the full extension to be activated but don't require the CLI. This suite is not run against multiple versions of the CLI in CI.
102-
* `test/vscode-tests/no-workspace` tests: These are intended to cover functionality around not having a workspace. The extension is not activated in these tests.
103-
* `test/vscode-tests/minimal-workspace` tests: These are intended to cover functionality that need a workspace but don't require the full extension to be activated.
104-
* CLI integration tests: these live in `test/vscode-tests/cli-integration`
105-
* These tests are intended to cover functionality that is related to the integration between the CodeQL CLI and the extension. These tests are run against each supported versions of the CLI in CI.
106-
107-
The CLI integration tests require an instance of the CodeQL CLI to run so they will require some extra setup steps. When adding new tests to our test suite, please be mindful of whether they need to be in the cli-integration folder. If the tests don't depend on the CLI, they are better suited to being a VSCode integration test.
108-
109-
Any test data you're using (sample projects, config files, etc.) must go in a `test/vscode-tests/*/data` directory. When you run the tests, the test runner will copy the data directory to `out/vscode-tests/*/data`.
110-
111-
#### Running the tests
112-
113-
Pre-requisites:
114-
1. Run `npm run build`.
115-
2. You will need to have `npm run watch` running in the background.
116-
117-
##### 1. From the terminal
118-
119-
Then, from the `extensions/ql-vscode` directory, use the appropriate command to run the tests:
120-
121-
* Unit tests: `npm run test:unit`
122-
* View Tests: `npm test:view`
123-
* VSCode integration tests: `npm run test:vscode-integration`
124-
125-
###### CLI integration tests
126-
127-
The CLI integration tests require the CodeQL standard libraries in order to run so you will need to clone a local copy of the `github/codeql` repository.
128-
129-
1. Set the `TEST_CODEQL_PATH` environment variable: running from a terminal, you _must_ set the `TEST_CODEQL_PATH` variable to point to a checkout of the `github/codeql` repository. The appropriate CLI version will be downloaded as part of the test.
130-
131-
2. Run your test command:
132-
133-
```shell
134-
cd extensions/ql-vscode && npm run test:cli-integration
135-
```
136-
137-
##### 2. From VSCode
138-
139-
Alternatively, you can run the tests inside of VSCode. There are several VSCode launch configurations defined that run the unit and integration tests.
140-
141-
You will need to run tests using a task from inside of VS Code, under the "Run and Debug" view:
142-
143-
* Unit tests: run the _Launch Unit Tests - React_ task
144-
* View Tests: run the _Launch Unit Tests_ task
145-
* VSCode integration tests: run the _Launch Unit Tests - No Workspace_ and _Launch Unit Tests - Minimal Workspace_ tasks
146-
147-
###### CLI integration tests
148-
149-
The CLI integration tests require the CodeQL standard libraries in order to run so you will need to clone a local copy of the `github/codeql` repository.
150-
151-
1. Set the `TEST_CODEQL_PATH` environment variable: running from a terminal, you _must_ set the `TEST_CODEQL_PATH` variable to point to a checkout of the `github/codeql` repository. The appropriate CLI version will be downloaded as part of the test.
152-
153-
2. Set the codeql path in VSCode's launch configuration: open `launch.json` and under the _Launch Integration Tests - With CLI_ section, uncomment the `"${workspaceRoot}/../codeql"` line. If you've cloned the `github/codeql` repo to a different path, replace the value with the correct path.
154-
155-
3. Run the VSCode task from the "Run and Debug" view called _Launch Integration Tests - With CLI_.
156-
157-
#### Running a single test
158-
159-
##### 1. From the terminal
160-
161-
The easiest way to run a single test is to change the `it` of the test to `it.only` and then run the test command with some additional options
162-
to only run tests for this specific file. For example, to run the test `test/vscode-tests/cli-integration/run-queries.test.ts`:
163-
164-
```shell
165-
npm run test:cli-integration -- --runTestsByPath test/vscode-tests/cli-integration/run-queries.test.ts
166-
```
167-
168-
You can also use the `--testNamePattern` option to run a specific test within a file. For example, to run the test `test/vscode-tests/cli-integration/run-queries.test.ts`:
169-
170-
```shell
171-
npm run test:cli-integration -- --runTestsByPath test/vscode-tests/cli-integration/run-queries.test.ts --testNamePattern "should create a QueryEvaluationInfo"
172-
```
173-
174-
##### 2. From VSCode
175-
176-
Alternatively, you can run a single test inside VSCode. To do so, install the [Jest Runner](https://marketplace.visualstudio.com/items?itemName=firsttris.vscode-jest-runner) extension. Then,
177-
you will have quicklinks to run a single test from within test files. To run a single unit or integration test, click the "Run" button. Debugging a single test is currently only supported
178-
for unit tests by default. To debug integration tests, open the `.vscode/settings.json` file and uncomment the `jestrunner.debugOptions` lines. This will allow you to debug integration tests.
179-
Please make sure to revert this change before committing; with this setting enabled, it is not possible to debug unit tests.
180-
181-
Without the Jest Runner extension, you can also use the "Launch Selected Unit Test (vscode-codeql)" launch configuration to run a single unit test.
182-
183-
#### Using a mock GitHub API server
184-
185-
Multi-Repo Variant Analyses (MRVA) rely on the GitHub API. In order to make development and testing easy, we have functionality that allows us to intercept requests to the GitHub API and provide mock responses.
186-
187-
##### Using a pre-recorded test scenario
188-
189-
To run a mock MRVA scenario, follow these steps:
190-
1. Enable the mock GitHub API server by adding the following in your VS Code user settings (which can be found by running the `Preferences: Open User Settings (JSON)` VS Code command):
191-
```json
192-
"codeQL.mockGitHubApiServer": {
193-
"enabled": true
194-
}
195-
```
196-
197-
1. Run the `CodeQL: Mock GitHub API Server: Load Scenario` command from the command pallet, and choose one of the scenarios to load.
198-
1. Execute a normal MRVA. At this point you should see the scenario being played out, rather than an actual MRVA running.
199-
1. Once you're done, you can stop using the mock scenario with `CodeQL: Mock GitHub API Server: Unload Scenario`
200-
201-
If you want to replay the same scenario you should unload and reload it so requests are replayed from the start.
202-
203-
##### Recording a new test scenario
204-
To record a new mock MRVA scenario, follow these steps:
205-
206-
1. Enable the mock GitHub API server by adding the following in your VS Code user settings (which can be found by running the `Preferences: Open User Settings (JSON)` VS Code command):
207-
```json
208-
"codeQL.mockGitHubApiServer": {
209-
"enabled": true
210-
}
211-
```
212-
213-
1. Run the `CodeQL: Mock GitHub API Server: Start Scenario Recording` VS Code command from the command pallet.
214-
1. Execute a normal MRVA.
215-
1. Once what you wanted to record is done (e.g. the MRVA has finished), then run the `CodeQL: Mock GitHub API Server: Save Scenario` command from the command pallet.
216-
1. The scenario should then be available for replaying.
217-
218-
If you want to cancel recording, run the `CodeQL: Mock GitHub API Server: Cancel Scenario Recording` command.
219-
220-
Once the scenario has been recorded, it's often useful to remove some of the requests to speed up the replay, particularly ones that fetch the variant analysis status. Once some of the request files have manually been removed, the [fix-scenario-file-numbering script](./extensions/ql-vscode/scripts/fix-scenario-file-numbering.ts) can be used to update the number of the files. See the script file for details on how to use.
221-
222-
#### Scenario data location
223-
224-
Pre-recorded scenarios are stored in `./src/mocks/scenarios`. However, it's possible to configure the location, by setting the `codeQL.mockGitHubApiServer.scenariosPath` configuration property in the VS Code user settings.
96+
Information about testing can be found [here](./docs/testing.md).
22597

22698
## Resources
22799

‎docs/testing.md‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
# Testing
2+
3+
We have several types of tests:
4+
5+
* Unit tests: these live in the `tests/unit-tests/` directory
6+
* View tests: these live in `src/view/variant-analysis/__tests__/`
7+
* VSCode integration tests:
8+
* `test/vscode-tests/activated-extension` tests: These are intended to cover functionality that require the full extension to be activated but don't require the CLI. This suite is not run against multiple versions of the CLI in CI.
9+
* `test/vscode-tests/no-workspace` tests: These are intended to cover functionality around not having a workspace. The extension is not activated in these tests.
10+
* `test/vscode-tests/minimal-workspace` tests: These are intended to cover functionality that need a workspace but don't require the full extension to be activated.
11+
* CLI integration tests: these live in `test/vscode-tests/cli-integration`
12+
* These tests are intended to cover functionality that is related to the integration between the CodeQL CLI and the extension. These tests are run against each supported versions of the CLI in CI.
13+
14+
The CLI integration tests require an instance of the CodeQL CLI to run so they will require some extra setup steps. When adding new tests to our test suite, please be mindful of whether they need to be in the cli-integration folder. If the tests don't depend on the CLI, they are better suited to being a VSCode integration test.
15+
16+
Any test data you're using (sample projects, config files, etc.) must go in a `test/vscode-tests/*/data` directory. When you run the tests, the test runner will copy the data directory to `out/vscode-tests/*/data`.
17+
18+
## Running the tests
19+
20+
Pre-requisites:
21+
1. Run `npm run build`.
22+
2. You will need to have `npm run watch` running in the background.
23+
24+
### 1. From the terminal
25+
26+
Then, from the `extensions/ql-vscode` directory, use the appropriate command to run the tests:
27+
28+
* Unit tests: `npm run test:unit`
29+
* View Tests: `npm test:view`
30+
* VSCode integration tests: `npm run test:vscode-integration`
31+
32+
#### CLI integration tests
33+
34+
The CLI integration tests require the CodeQL standard libraries in order to run so you will need to clone a local copy of the `github/codeql` repository.
35+
36+
1. Set the `TEST_CODEQL_PATH` environment variable: running from a terminal, you _must_ set the `TEST_CODEQL_PATH` variable to point to a checkout of the `github/codeql` repository. The appropriate CLI version will be downloaded as part of the test.
37+
38+
2. Run your test command:
39+
40+
```shell
41+
cd extensions/ql-vscode && npm run test:cli-integration
42+
```
43+
44+
### 2. From VSCode
45+
46+
Alternatively, you can run the tests inside of VSCode. There are several VSCode launch configurations defined that run the unit and integration tests.
47+
48+
You will need to run tests using a task from inside of VS Code, under the "Run and Debug" view:
49+
50+
* Unit tests: run the _Launch Unit Tests - React_ task
51+
* View Tests: run the _Launch Unit Tests_ task
52+
* VSCode integration tests: run the _Launch Unit Tests - No Workspace_ and _Launch Unit Tests - Minimal Workspace_ tasks
53+
54+
#### CLI integration tests
55+
56+
The CLI integration tests require the CodeQL standard libraries in order to run so you will need to clone a local copy of the `github/codeql` repository.
57+
58+
1. Set the `TEST_CODEQL_PATH` environment variable: running from a terminal, you _must_ set the `TEST_CODEQL_PATH` variable to point to a checkout of the `github/codeql` repository. The appropriate CLI version will be downloaded as part of the test.
59+
60+
2. Set the codeql path in VSCode's launch configuration: open `launch.json` and under the _Launch Integration Tests - With CLI_ section, uncomment the `"${workspaceRoot}/../codeql"` line. If you've cloned the `github/codeql` repo to a different path, replace the value with the correct path.
61+
62+
3. Run the VSCode task from the "Run and Debug" view called _Launch Integration Tests - With CLI_.
63+
64+
## Running a single test
65+
66+
### 1. From the terminal
67+
68+
The easiest way to run a single test is to change the `it` of the test to `it.only` and then run the test command with some additional options
69+
to only run tests for this specific file. For example, to run the test `test/vscode-tests/cli-integration/run-queries.test.ts`:
70+
71+
```shell
72+
npm run test:cli-integration -- --runTestsByPath test/vscode-tests/cli-integration/run-queries.test.ts
73+
```
74+
75+
You can also use the `--testNamePattern` option to run a specific test within a file. For example, to run the test `test/vscode-tests/cli-integration/run-queries.test.ts`:
76+
77+
```shell
78+
npm run test:cli-integration -- --runTestsByPath test/vscode-tests/cli-integration/run-queries.test.ts --testNamePattern "should create a QueryEvaluationInfo"
79+
```
80+
81+
### 2. From VSCode
82+
83+
Alternatively, you can run a single test inside VSCode. To do so, install the [Jest Runner](https://marketplace.visualstudio.com/items?itemName=firsttris.vscode-jest-runner) extension. Then,
84+
you will have quicklinks to run a single test from within test files. To run a single unit or integration test, click the "Run" button. Debugging a single test is currently only supported
85+
for unit tests by default. To debug integration tests, open the `.vscode/settings.json` file and uncomment the `jestrunner.debugOptions` lines. This will allow you to debug integration tests.
86+
Please make sure to revert this change before committing; with this setting enabled, it is not possible to debug unit tests.
87+
88+
Without the Jest Runner extension, you can also use the "Launch Selected Unit Test (vscode-codeql)" launch configuration to run a single unit test.
89+
90+
## Using a mock GitHub API server
91+
92+
Multi-Repo Variant Analyses (MRVA) rely on the GitHub API. In order to make development and testing easy, we have functionality that allows us to intercept requests to the GitHub API and provide mock responses.
93+
94+
### Using a pre-recorded test scenario
95+
96+
To run a mock MRVA scenario, follow these steps:
97+
1. Enable the mock GitHub API server by adding the following in your VS Code user settings (which can be found by running the `Preferences: Open User Settings (JSON)` VS Code command):
98+
```json
99+
"codeQL.mockGitHubApiServer": {
100+
"enabled": true
101+
}
102+
```
103+
104+
1. Run the `CodeQL: Mock GitHub API Server: Load Scenario` command from the command pallet, and choose one of the scenarios to load.
105+
1. Execute a normal MRVA. At this point you should see the scenario being played out, rather than an actual MRVA running.
106+
1. Once you're done, you can stop using the mock scenario with `CodeQL: Mock GitHub API Server: Unload Scenario`
107+
108+
If you want to replay the same scenario you should unload and reload it so requests are replayed from the start.
109+
110+
### Recording a new test scenario
111+
To record a new mock MRVA scenario, follow these steps:
112+
113+
1. Enable the mock GitHub API server by adding the following in your VS Code user settings (which can be found by running the `Preferences: Open User Settings (JSON)` VS Code command):
114+
```json
115+
"codeQL.mockGitHubApiServer": {
116+
"enabled": true
117+
}
118+
```
119+
120+
1. Run the `CodeQL: Mock GitHub API Server: Start Scenario Recording` VS Code command from the command pallet.
121+
1. Execute a normal MRVA.
122+
1. Once what you wanted to record is done (e.g. the MRVA has finished), then run the `CodeQL: Mock GitHub API Server: Save Scenario` command from the command pallet.
123+
1. The scenario should then be available for replaying.
124+
125+
If you want to cancel recording, run the `CodeQL: Mock GitHub API Server: Cancel Scenario Recording` command.
126+
127+
Once the scenario has been recorded, it's often useful to remove some of the requests to speed up the replay, particularly ones that fetch the variant analysis status. Once some of the request files have manually been removed, the [fix-scenario-file-numbering script](../extensions/ql-vscode/scripts/fix-scenario-file-numbering.ts) can be used to update the number of the files. See the script file for details on how to use.
128+
129+
### Scenario data location
130+
131+
Pre-recorded scenarios are stored in `./src/mocks/scenarios`. However, it's possible to configure the location, by setting the `codeQL.mockGitHubApiServer.scenariosPath` configuration property in the VS Code user settings.

0 commit comments

Comments
 (0)