22
33This document describes the manual test plan for the QL extension for Visual Studio Code.
44
5- The plan will be executed manually to start with but the goal is to eventually automate parts of the process (based on
5+ The plan will be executed manually to start with but the goal is to eventually automate parts of the process (based on
66effort vs value basis).
77
8- #### What this doesn't cover
8+ ## What this doesn't cover
9+
910We don't need to test features (and permutations of features) that are covered by automated tests.
1011
11- ### Before releasing the VS Code extension
12+ ## Before releasing the VS Code extension
13+
1214- Go through the required test cases listed below
13- - Check major PRs since the previous release for specific one-off things to test. Based on that, you might want to
15+ - Check major PRs since the previous release for specific one-off things to test. Based on that, you might want to
1416choose to go through some of the Optional Test Cases.
1517- Run a query using the existing version of the extension (to generate an "old" query history item)
1618
1719## Required Test Cases
1820
21+ ### Pre-requisites
22+
23+ - Flip the ` codeQL.canary ` flag. This will enable MRVA in the extension.
24+
1925### Test Case 1: MRVA - Running a problem path query and viewing results
2026
21271 . Open the [ UnsafeJQueryPlugin query] ( https://github.com/github/codeql/blob/main/javascript/ql/src/Security/CWE-079/UnsafeJQueryPlugin.ql ) .
22282 . Run a MRVA against the following repo list:
23- ```
24- {
25- "name": "test-repo-list",
26- "repositories": [
27- "angular-cn/ng-nice",
28- "apache/hadoop",
29- "apache/hive"
30- ]
31- }
32- ```
29+
30+ ``` json
31+ {
32+ "name" : " test-repo-list" ,
33+ "repositories" : [
34+ " angular-cn/ng-nice" ,
35+ " apache/hadoop" ,
36+ " apache/hive"
37+ ]
38+ }
39+ ```
40+
33413 . Check that a notification message pops up and the results view is opened.
34424 . Check the query history. It should:
3543 - Show that an item has been added to the query history
3644 - The item should be marked as "in progress".
37455 . Once the query starts:
38- - Check the results view
39- - Check the code paths view, including the code paths drop down menu.
46+ - Check the results view
47+ - Check the code paths view, including the code paths drop down menu.
4048 - Check that the repository filter box works
4149 - Click links to files/locations on GitHub
4250 - Check that the query history item is updated to show the number of results
@@ -70,7 +78,7 @@ choose to go through some of the Optional Test Cases.
70781 . Click a history item (for MRVA):
7179 - Check that exporting results works
7280 - Check that sorting results works
73- - Check that copying repo lists works
81+ - Check that copying repo lists works
74822 . Open the query results directory:
7583 - Check that the correct directory is opened and there are results in it
76843 . View logs
@@ -80,12 +88,12 @@ choose to go through some of the Optional Test Cases.
8088
8189Run one of the above MRVAs, but cancel it from within VS Code:
8290- Check that the query is canceled and the query history item is updated.
83- - Check that the workflow run is also canceled.
91+ - Check that the workflow run is also canceled.
8492- Check that any available results are visible in VS Code.
8593
86- ### Test Case 6: MRVA - Change to a different colour theme
94+ ### Test Case 6: MRVA - Change to a different colour theme
8795
88- Open one of the above MRVAs, try changing to a different colour theme and check that everything looks sensible.
96+ Open one of the above MRVAs, try changing to a different colour theme and check that everything looks sensible.
8997Are there any components that are not showing up?
9098
9199## Optional Test Cases
@@ -95,24 +103,27 @@ These are mostly aimed at MRVA, but some of them are also applicable to non-MRVA
95103### Selecting repositories to run on
96104
97105#### Test case 1: Running a query on a single repository
98- 1 . When the repository exists and is public
99- 1 . Has a CodeQL database for the correct language
100- 2 . Has a CodeQL database for another language
106+
107+ 1 . When the repository exists and is public
108+ 1 . Has a CodeQL database for the correct language
109+ 2 . Has a CodeQL database for another language
101110 3 . Does not have any CodeQL databases
1021112 . When the repository exists and is private
103112 1 . Is accessible and has a CodeQL database
104113 2 . Is not accessible
1051143 . When the repository does not exist
106115
107116#### Test case 2: Running a query on a custom repository list
117+
1081181 . The repository list is non-empty
109- 1 . All repositories in the list have a CodeQL database
119+ 1 . All repositories in the list have a CodeQL database
110120 2 . Some but not all repositories in the list have a CodeQL database
111121 3 . No repositories in the list have a CodeQL database
1121222 . The repository list is empty
113123
114124#### Test case 3: Running a query on all repositories in an organization
115- 1 . The org exists
125+
126+ 1 . The org exists
116127 1 . The org contains repositories that have CodeQL databases
117128 2 . The org contains repositories of the right language but without CodeQL databases
118129 3 . The org contains repositories not of the right language
@@ -122,20 +133,25 @@ These are mostly aimed at MRVA, but some of them are also applicable to non-MRVA
122133### Using different types of controller repos
123134
124135#### Test case 1: Running a query when the controller repository is public
136+
1251371 . Can run queries on public repositories
1261382 . Can not run queries on private repositories
127139
128140#### Test case 2: Running a query when the controller repository is private
141+
1291421 . Can run queries on public repositories
1301432 . Can run queries on private repositories
131144
132145#### Test case 3: Running a query when the controller repo exists but you do not have write access
146+
1331471 . Cannot run queries
134148
135149#### Test case 4: Running a query when the controller repo doesn’t exist
150+
1361511 . Cannot run queries
137152
138153#### Test case 5: Running a query when the "config field" for the controller repo is not set
154+
1391551 . Cannot run queries
140156
141157### Query History
@@ -146,6 +162,7 @@ The first test case specifies actions that you can do when the query is first ru
146162with this since it has quite a limited number of actions you can do.
147163
148164#### Test case 1: When variant analysis state is "pending"
165+
1491661 . Starts monitoring variant analysis
1501672 . Cannot open query history item
1511683 . Can delete a query history item
@@ -156,8 +173,8 @@ with this since it has quite a limited number of actions you can do.
156173 2 . By query date
157174 3 . By result count
1581755 . Cannot open query directory
159- 6 . Can open query that produced these results
160- 1 . When the file still exists and has not moved
176+ 6 . Can open query that produced these results
177+ 1 . When the file still exists and has not moved
161178 2 . When the file does not exist
1621797 . Cannot view logs
1631808 . Cannot copy repository list
@@ -167,6 +184,7 @@ with this since it has quite a limited number of actions you can do.
16718412 . Cannot cancel analysis
168185
169186#### Test case 2: When the variant analysis state is not "pending"
187+
1701881 . Query history is loaded when VSCode starts
1711892 . Handles when action workflow was canceled while VSCode was closed
1721903 . Can open query history item
@@ -200,12 +218,14 @@ with this since it has quite a limited number of actions you can do.
200218 4 . A popup allows you to open the directory
201219
202220#### Test case 3: When variant analysis state is "in_progress"
221+
2032221 . Starts monitoring variant analysis
204- 1 . Ready results are downloaded
205- 2 . Can cancel analysis
223+ 1 . Ready results are downloaded
224+ 2 . Can cancel analysis
206225 1 . Causes the actions run to be canceled
207226
208227#### Test case 4: When variant analysis state is in final state ("succeeded"/"failed"/"canceled")
228+
2092291 . Stops monitoring variant analysis
210230 1 . All results are downloaded if state is succeeded
211231 2 . Otherwise, ready results are downloaded, if any are available
@@ -216,16 +236,18 @@ with this since it has quite a limited number of actions you can do.
216236This requires running a MRVA query and seeing the results view.
217237
218238#### Test case 1: When variant analysis state is "pending"
239+
2192401 . Can open a results view
2202412 . Results view opens automatically
221242 - When starting variant analysis run
222243 - When VSCode opens (if view was open when VSCode was closed)
2232443 . Results view is empty
224245
225246#### Test case 2: When variant analysis state is not "pending"
247+
2262481 . Can open a results view
2272492 . Results view opens automatically
228- 1 . When starting variant analysis run
250+ 1 . When starting variant analysis run
229251 2 . When VSCode opens (if view was open when VSCode was closed)
2302523 . Can copy repository list
231253 1 . Text is copied to clipboard
@@ -236,43 +258,45 @@ This requires running a MRVA query and seeing the results view.
2362586 . Can open query file
237259 1 . When the file still exists and has not moved
238260 2 . When the file does not exist
239- 7 . Can open query text
240- 8 . Can sort repos
241- 1 . By name
242- 2 . By results
243- 3 . By stars
261+ 7 . Can open query text
262+ 8 . Can sort repos
263+ 1 . By name
264+ 2 . By results
265+ 3 . By stars
244266 4 . By last updated
2452679 . Can filter repos
246- 10 . Shows correct statistics
247- 1 . Total number of results
248- 2 . Total number of repositories
268+ 10 . Shows correct statistics
269+ 1 . Total number of results
270+ 2 . Total number of repositories
249271 3 . Duration
250- 11 . Can see live results
272+ 11 . Can see live results
251273 1 . Results appear in extension as soon as each query is completed
25227412 . Can view interpreted results (i.e. for a "problem" query)
253- 1 . Can view non-path results
275+ 1 . Can view non-path results
254276 2 . Can view code paths for "path-problem" queries
25527713 . Can view raw results (i.e. for a non "problem" query)
256278 1 . Renders a table
257- 14 . Can see skipped repositories
258- 1 . Can see repos with no db in a tab
259- 1 . Shown warning that explains the tab
279+ 14 . Can see skipped repositories
280+ 1 . Can see repos with no db in a tab
281+ 1 . Shown warning that explains the tab
260282 2 . Can see repos with no access in a tab
261- 1 . Shown warning that explains the tab
283+ 1 . Shown warning that explains the tab
262284 3 . Only shows tab when there are skipped repos
263- 15 . Result downloads
264- 1 . All results are downloaded automatically
285+ 15 . Result downloads
286+ 1 . All results are downloaded automatically
265287 2 . Download status is indicated by a spinner (Not currently any indication of progress beyond "downloading" and "not downloading")
266- 3 . Only 3 items are downloaded at a time
267- 4 . Results for completed queries are still downloaded when
268- 1 . Some but not all queries failed
288+ 3 . Only 3 items are downloaded at a time
289+ 4 . Results for completed queries are still downloaded when
290+ 1 . Some but not all queries failed
269291 2 . The variant analysis was canceled after some queries completed
270292
271293#### Test case 3: When variant analysis state is in "succeeded" state
294+
2722951 . Can view logs
273- 2 . All results are downloaded
296+ 2 . All results are downloaded
274297
275298#### Test case 4: When variant analysis is in "failed" or "canceled" state
299+
2763001 . Can view logs
2773011 . Results for finished queries are still downloaded.
278302
@@ -301,14 +325,17 @@ This requires running a MRVA query and seeing the results view.
3013251 . Collapse/expand tree nodes
302326
303327Error cases that trigger an error notification:
304- 1 . Try to add a list with a name that already exists
328+
329+ 1 . Try to add a list with a name that already exists
3053301 . Try to add a top-level database that already exists
3063311 . Try to add a database in a list that already exists in the list
307332
308333Error cases that show an error in the panel (and only the edit button should be visible):
334+
3093351 . Edit the db config file directly and save invalid JSON
3103361 . Edit the db config file directly and save valid JSON but invalid config (e.g. add an unknown property)
311- 1 . Edit the db config file directly and save two lists with the same name
337+ 1 . Edit the db config file directly and save two lists with the same name
312338
313339Cases where there the welcome view is shown:
314- 1 . No controller repo is set in the user's settings JSON.
340+
341+ 1 . No controller repo is set in the user's settings JSON.
0 commit comments