Skip to content

docs: document CI test suite registration requirement #3943

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andygrove wants to merge 4 commits into
base: main
Choose a base branch
from
+45 −1
Open

docs: document CI test suite registration requirement #3943

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e67cbf2
docs: document CI test suite registration requirement
andygrove Apr 14, 2026
eeaf356
docs: mention pr_missing_suites CI check that detects missing suites
andygrove Apr 14, 2026
3fc6536
fix: use synchronize instead of edited trigger for missing suites check
andygrove Apr 14, 2026
46d6941
ci: run missing suites check on pushes to main
andygrove Apr 14, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/pr_missing_suites.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,11 @@
name: Check that all test suites are added to PR workflows

on:
push:
branches:
- main
pull_request:
types: [opened, edited, reopened]
types: [opened, synchronize, reopened]

jobs:
check-missing-suites:
Expand Down
41 changes: 41 additions & 0 deletions docs/source/contributor-guide/development.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,47 @@ make test-rust
make test-jvm
```

### 5. Register New Test Suites in CI

Comet's CI does not automatically discover test suites. Instead, test suites are explicitly listed
in the GitHub Actions workflow files so they can be grouped by category and run as separate parallel
jobs. This reduces overall CI time.

If you add a new Scala test suite, you must add it to the `suite` matrix in **both** workflow files:

- `.github/workflows/pr_build_linux.yml`
- `.github/workflows/pr_build_macos.yml`

Each file contains a `suite` matrix with named groups such as `fuzz`, `shuffle`, `parquet`, `csv`,
`exec`, `expressions`, and `sql`. Add your new suite's fully qualified class name to the
appropriate group. For example, if you add a new expression test suite, add it to the
`expressions` group:

```yaml
- name: "expressions"
value: |
org.apache.comet.CometExpressionSuite
# ... existing suites ...
org.apache.comet.YourNewExpressionSuite # <-- add here
```

Choose the group that best matches the area your test covers:

| Group | Covers |
| ------------- | --------------------------------------------------------- |
| `fuzz` | Fuzz testing and data generation |
| `shuffle` | Shuffle operators and related exchange behavior |
| `parquet` | Parquet read/write and native reader tests |
| `csv` | CSV native read tests |
| `exec` | Execution operators, joins, aggregates, plan rules, TPC-* |
| `expressions` | Expression evaluation, casts, and SQL file tests |
| `sql` | SQL-level behavior tests |

**Important:** The suite lists in both workflow files must stay in sync. A separate CI check
(`.github/workflows/pr_missing_suites.yml`) runs `dev/ci/check-suites.py` on every pull request.
It scans for all `*Suite.scala` files in the repository and verifies that each one appears in both
workflow files. If any suite is missing, this check will fail and block the PR.

### Pre-PR Summary

```sh
Expand Down
Loading