feat(scanner): enable SkillSpector LLM semantic pass (Anthropic Sonnet)

[DevelopmentCats]

What this does

Flips the scheduled scan from --no-llm to SkillSpector's two-stage analyser (static rules + LLM semantic pass). Upstream's published precision goes from ~70% to ~87% by filtering context-aware false positives.

Provider is Anthropic (claude-sonnet-4-6). Sonnet is roughly 5x cheaper than the upstream default Opus and is well matched to the finding-classification work the LLM pass actually does.

Stack

Stacked on PR #1. Merge that first, then this can target main.

Setup before merge

1. Add ANTHROPIC_API_KEY as a repo secret (Settings > Secrets and variables > Actions).
2. Apply the workflow diff to .github/workflows/scan.yaml (see collapsible below). The Coder Agents app on this repo doesn't have workflows: write, so the bot couldn't commit that file. Either grant the permission and ping me to re-push, or paste the diff yourself.
3. After merge: Actions > scan > Run workflow.

Graceful fallback

If the secret isn't set, the workflow emits a warning and runs --no-llm. A fresh fork keeps producing valid (lower-precision) scans without any setup.

Expected impact on the five in-tree skills

Skill	static-only today	with LLM on
coder/coder-modules	clean (0)	clean
coder/coder-templates	clean (0)	clean
coder/modules	clean (0)	clean
coder/templates	clean (10)	clean
coder/setup	malicious (100)	still high, findings list shrinks

Bringing coder/setup below suspicious needs the Phase 3 permissions-manifest layer (different PR). This change is the precision prerequisite, not the false-positive fix.

Workflow file diff (to paste into .github/workflows/scan.yaml)

Three edits inside the scan job:

1. Add new step after Verify skill path exists:

      - name: Determine LLM mode
        id: llm
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          set -euo pipefail
          if [[ -n "${ANTHROPIC_API_KEY:-}" ]]; then
            echo "extra_flags=" >> "$GITHUB_OUTPUT"
            echo "SkillSpector LLM mode: enabled (anthropic provider)." >&2
          else
            echo "extra_flags=--no-llm" >> "$GITHUB_OUTPUT"
            echo "::warning::ANTHROPIC_API_KEY secret not set; SkillSpector will run with --no-llm. See README \"One-time setup\" to enable the LLM semantic pass."
          fi

2. Replace the SkillSpector (JSON) step:

      - name: SkillSpector (JSON)
        if: steps.path_check.outputs.drift == 'false'
        continue-on-error: true
        env:
          SKILLSPECTOR_PROVIDER: anthropic
          SKILLSPECTOR_MODEL: claude-sonnet-4-6
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          mkdir -p out
          skillspector scan "source/${{ matrix.skill_path }}" \
            ${{ steps.llm.outputs.extra_flags }} \
            --format json \
            --output "out/skillspector.json" || true

3. Same swap for SkillSpector (SARIF):

      - name: SkillSpector (SARIF)
        if: steps.path_check.outputs.drift == 'false'
        continue-on-error: true
        env:
          SKILLSPECTOR_PROVIDER: anthropic
          SKILLSPECTOR_MODEL: claude-sonnet-4-6
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          mkdir -p out
          skillspector scan "source/${{ matrix.skill_path }}" \
            ${{ steps.llm.outputs.extra_flags }} \
            --format sarif \
            --output "out/skillspector.sarif" || true

Decision log

• Anthropic over NVIDIA Build: Coder already has the billing relationship and the credential is one secret away. No second vendor signup.
• Sonnet over Opus: SkillSpector's LLM pass is per-finding intent classification, not long-form reasoning. Sonnet at ~5x lower cost is the better cost/quality choice for periodic scans.
• Hardcoded SKILLSPECTOR_PROVIDER in workflow: simpler than reading from config.yaml at runtime. One line to change in two places when we add a second provider in practice.
• Graceful --no-llm fallback: a fresh fork should produce something useful immediately, not 404 the publish pipeline because the operator hasn't added the secret yet.
• No CI change: ci.yaml uses inline pytest fixtures and never invokes skillspector live, so no inference cost on PR review.
• No schema or verdict-math change: LLM mode shifts which findings reach the verdict, not how the verdict is computed. CALIBRATION.md walks through this explicitly.

This PR was prepared with help from Coder Agents.

[Copilot]

Pull request overview

Updates the scanner configuration and documentation to support running NVIDIA SkillSpector with an optional LLM semantic pass (with a fallback to static-only mode), as part of the scheduled scan pipeline described in this repo.

Changes:

• Extend config.yaml with a scanners.skillspector.llm configuration block and make scanners.skillspector.flags empty by default.
• Document the LLM semantic pass behavior, expected precision delta, and repo one-time setup steps in README.md and docs/CALIBRATION.md.
• Shift responsibility for driving --no-llm to the scheduled workflow (though the workflow change itself is not included in this PR’s diff).

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 7 comments.

File	Description
README.md	Updates architecture and adds one-time setup guidance, including LLM credential setup.
docs/CALIBRATION.md	Adds an “LLM semantic pass” section explaining expected effects and limitations.
config.yaml	Adds scanners.skillspector.llm block and removes the default --no-llm flag from config-driven flags.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.