Skip to content

Prepare for rolling out skipping computing file coverage information on PRs#3562

Open
henrymercer wants to merge 5 commits intomainfrom
henrymercer/skip-file-coverage-rollout
Open

Prepare for rolling out skipping computing file coverage information on PRs#3562
henrymercer wants to merge 5 commits intomainfrom
henrymercer/skip-file-coverage-rollout

Conversation

@henrymercer
Copy link
Contributor

Prepare for rolling out skipping computing file coverage information on PRs to speed up analysis.

  • Gate the feature behind a CLI feature that updates the logs to account for file coverage information no longer being expected.
  • Add a repository property that can be used to continue computing file coverage information on PRs.
  • Add a telemetry diagnostic so we can see whether this is being used.
  • Remove the restriction to the github org.

Note that the CLI tools feature flag is not yet released, even as a nightly. Though given it that CLI feature only modifies logs, I don't think we need to wait for a release before merging this PR.

Risk assessment

For internal use only. Please select the risk level of this change:

  • Low risk: Changes are fully under feature flags, or have been fully tested and validated in pre-production environments and are highly observable, or are documentation or test only.

Which use cases does this change impact?

Workflow types:

  • Advanced setup - Impacts users who have custom CodeQL workflows.
  • Managed - Impacts users with dynamic workflows (Default Setup, Code Quality, ...).

Products:

  • Code Scanning - The changes impact analyses when analysis-kinds: code-scanning.

Environments:

  • Dotcom - Impacts CodeQL workflows on github.com and/or GitHub Enterprise Cloud with Data Residency.

How did/will you validate this change?

  • Unit tests - I am depending on unit test coverage (i.e. tests in .test.ts files).

If something goes wrong after this change is released, what are the mitigation and rollback strategies?

  • Feature flags - All new or changed code paths can be fully disabled with corresponding feature flags.

How will you know if something goes wrong after this change is released?

  • Telemetry - I rely on existing telemetry or have made changes to the telemetry.
    • Alerts - New or existing monitors will trip if something goes wrong with this change.

Are there any special considerations for merging or releasing this change?

  • No special considerations - This change can be merged at any time.

Merge / deployment checklist

  • Confirm this change is backwards compatible with existing workflows.
  • Consider adding a changelog entry for this change.
  • Confirm the readme and docs have been updated if necessary.

Copilot AI review requested due to automatic review settings March 9, 2026 17:20
@henrymercer henrymercer requested a review from a team as a code owner March 9, 2026 17:20
@github-actions github-actions bot added the size/S Should be easy to review label Mar 9, 2026
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Prepares a controlled rollout to skip computing file coverage information on PR analyses (to improve performance) by gating the behavior on a CodeQL CLI tools feature, adding a repository property opt-out, and emitting telemetry to observe usage.

Changes:

  • Add a new CodeQL CLI tools feature (suppressesMissingFileBaselineWarning) and gate SkipFileCoverageOnPrs behind it.
  • Allow org-owned repositories to opt back into file coverage on PRs via the github-codeql-file-coverage-on-prs repository property.
  • Add telemetry for when the repository property is used, plus update unit tests and changelog.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
src/tools-features.ts Adds a new CLI tools feature enum value used to gate the rollout.
src/feature-flags.ts Gates SkipFileCoverageOnPrs behind the new CLI tools feature.
src/feature-flags/properties.ts Adds and parses the new repository property github-codeql-file-coverage-on-prs.
src/init.ts Updates file-coverage enablement logic to support repository-property opt-in and uses CLI-gated feature evaluation.
src/init-action.ts Passes repository properties into config calculation and emits telemetry when repo property enables PR file coverage.
src/init.test.ts Updates unit tests for the revised file-coverage enablement logic and new opt-in behavior.
CHANGELOG.md Documents the experimental rollout and the repository-property opt-out mechanism.
lib/upload-sarif-action.js Generated JS output updated to reflect TS changes.
lib/upload-sarif-action-post.js Generated JS output updated to reflect TS changes.
lib/upload-lib.js Generated JS output updated to reflect TS changes.
lib/start-proxy-action.js Generated JS output updated to reflect TS changes.
lib/start-proxy-action-post.js Generated JS output updated to reflect TS changes.
lib/setup-codeql-action.js Generated JS output updated to reflect TS changes.
lib/resolve-environment-action.js Generated JS output updated to reflect TS changes.
lib/init-action.js Generated JS output updated to reflect TS changes.
lib/init-action-post.js Generated JS output updated to reflect TS changes.
lib/autobuild-action.js Generated JS output updated to reflect TS changes.
lib/analyze-action.js Generated JS output updated to reflect TS changes.
lib/analyze-action-post.js Generated JS output updated to reflect TS changes.

Copy link
Member

@mbg mbg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM as-is, but I have a couple of suggestions for tidying up the implementation. I also think we should add a page to docs.github.com for the available repository properties that are understood by the CodeQL Action.

- Fixed [a bug](https://github.com/github/codeql-action/issues/3555) which caused the CodeQL Action to fail loading repository properties if a "Multi select" repository property was configured for the repository. [#3557](https://github.com/github/codeql-action/pull/3557)
- Added an experimental change which skips collecting file coverage information on pull requests to improve analysis performance. File coverage information will still be computed on non-PR analyses.

Repositories owned by an organization can opt out of this change by creating a custom repository property with the name `github-codeql-file-coverage-on-prs` and the type "True/false", then setting this property to `true` in the repository's settings. For more information, see [Managing custom properties for repositories in your organization](https://docs.github.com/en/organizations/managing-organization-settings/managing-custom-properties-for-repositories-in-your-organization).
Copy link
Member

@mbg mbg Mar 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add documentation on docs.github.com about these repo properties. I have created an internal issue to track this.

repositoryNwo,
codeql,
features,
repositoryPropertiesResult.orElse({}),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: we have repositoryPropertiesResult.orElse({}) at least a couple of times here now. Consider adding const repositoryProperties = repositoryPropertiesResult.orElse({}); somewhere before the uses, or indeed modifying the code so that we never end up with the "result" here, but only the properties we loaded or {}.

Comment on lines +404 to +409
config.enableFileCoverageInformation &&
isAnalyzingPullRequest() &&
(await features.getValue(Feature.SkipFileCoverageOnPrs, codeql)) &&
repositoryPropertiesResult.orElse({})[
RepositoryPropertyName.FILE_COVERAGE_ON_PRS
] === true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This now duplicates most of getFileCoverageInformationEnabled. While we can't use getFileCoverageInformationEnabled as a shortcut here, consider whether you can move the code that emits this diagnostic into getFileCoverageInformationEnabled and end up with a cleaner solution in there?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/S Should be easy to review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants