From a21fe17d8ff2b405332fc02befd03ae93a284d71 Mon Sep 17 00:00:00 2001 From: DevCats Date: Mon, 22 Jun 2026 14:50:28 -0500 Subject: [PATCH 01/13] feat(config.yaml): add scanners.skillspector.llm block Document the new llm.provider config knob (default nv_build) and the workflow contract: empty flags + workflow appends --no-llm dynamically when the matching credential secret is unset. Removes --no-llm from the static flags list now that the workflow drives it. This commit was prepared with help from Coder Agents. --- config.yaml | 37 +++++++++++++++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/config.yaml b/config.yaml index b20462a..31be704 100644 --- a/config.yaml +++ b/config.yaml @@ -39,8 +39,41 @@ scanners: # so a bumper bot lives outside the loop until the upstream # publishes to PyPI and the pin can move into pyproject.toml. pin: "skillspector @ git+https://github.com/NVIDIA/SkillSpector.git@2eb844780ab163f01468ecf142c40a2ec0fcaec0" - flags: - - "--no-llm" + # Extra CLI flags passed to every SkillSpector invocation. Empty by + # default; the scan workflow appends --no-llm dynamically when the + # LLM credential secret is not set (see llm: block below). CI runs + # do not invoke SkillSpector live. + flags: [] + # SkillSpector ships a two-stage analyser: fast static rules followed + # by an optional LLM semantic pass. The LLM pass lifts precision + # from roughly 70% to roughly 87% per upstream docs by filtering + # context-aware false positives, classifying intent on prompt + # injection patterns, and producing human-readable explanations. + # + # The scheduled scan reads the credential matching the provider + # below from a repository secret. When the secret is configured, + # LLM mode is on. When the secret is missing, the workflow falls + # back to --no-llm automatically so a fresh fork is never broken + # by an unset secret. + # + # Provider options and the env var SkillSpector consumes: + # + # provider env var(s) + # nv_build NVIDIA_INFERENCE_KEY (free; build.nvidia.com) + # openai OPENAI_API_KEY (+ OPENAI_BASE_URL for AI gateways) + # anthropic ANTHROPIC_API_KEY + # anthropic_proxy ANTHROPIC_PROXY_API_KEY + ANTHROPIC_PROXY_ENDPOINT_URL + # + # Changing provider also requires updating the env block in + # .github/workflows/scan.yaml so the matching secret is wired in, + # and adding the secret under Settings > Secrets and variables > + # Actions. + llm: + provider: nv_build + # Empty model uses the provider's bundled default. Override here + # to pin a specific revision (e.g. "claude-opus-4-6" for + # provider=anthropic). + model: "" # Per-skill verdict policy. v1 has one input (SkillSpector risk_score). # When more scanners join the pipeline we add new threshold fields here From c3f42dbd21be10b86869a9249af54229e6fc2968 Mon Sep 17 00:00:00 2001 From: DevCats Date: Mon, 22 Jun 2026 14:53:56 -0500 Subject: [PATCH 02/13] docs(CALIBRATION.md): add LLM semantic pass section Document what flipping LLM mode on does (and does not do) to the verdict math, the precision delta we expect, and what to expect for the five in-tree skills. Adds "LLM provider changes" to the "When to revisit" list. This commit was prepared with help from Coder Agents. --- docs/CALIBRATION.md | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/docs/CALIBRATION.md b/docs/CALIBRATION.md index e4cf5a0..5091163 100644 --- a/docs/CALIBRATION.md +++ b/docs/CALIBRATION.md @@ -99,6 +99,42 @@ verdict: This avoids broadcasting the ~half-of-catalogue base rate that ClawHub measured. +## LLM semantic pass + +SkillSpector ships a two-stage analyser: fast static rules (the 64 +patterns SkillSpector documents) followed by an optional LLM semantic +pass. Upstream's published precision numbers are: + +- `--no-llm` (static only): high recall, moderate precision (~70%). + False positives on context-sensitive patterns are common; for + example, EA2 ("autonomous decision making") fires on prose that + documents safeguards as well as prose that bypasses them. +- Default (LLM on): ~87% precision. The LLM pass reads each finding's + surrounding context, classifies intent, filters context-aware false + positives, and writes a human-readable explanation that ships in the + per-finding output. + +The scheduled scan runs LLM mode when the workflow's chosen credential +secret (`NVIDIA_INFERENCE_KEY` for the default `nv_build` provider) is +configured. The fallback to `--no-llm` is automatic when the secret is +missing, so an unset secret on a fresh fork degrades the scan rather +than breaking it. + +The LLM pass does not affect the threshold math: SkillSpector's +`risk_score` is still a 0-100 weighted sum of rule hits, and the +51/81 cutoffs above still map directly to `HIGH` and `CRITICAL` bands. +It does affect which findings reach the verdict: false positives that +the LLM filters out no longer contribute to the score. Expect verdicts +to move down (or stay the same) when LLM mode flips on, not up. + +For the five existing in-tree skills, the static-only scan placed +`coder/setup` at 100 / `malicious`. With LLM mode on we expect the +findings list to shrink (the EA2 prose hits and the asset-path MP2 +hits should be filtered) but the score will still be high. Reducing +`coder/setup`'s verdict below `suspicious` requires the upcoming +permissions-manifest layer (Phase 3 of the v3 plan), not the LLM pass +alone. + ## What we did not change (and why) - We did not raise `suspicious_risk_score` above `51`. SkillSpector @@ -127,6 +163,9 @@ Re-run this analysis when any of: that shifts where its bands sit. The pinned commit in `config.yaml` protects us from drifting silently; a deliberate bump should walk through this doc. +- The LLM provider changes (e.g., moving from `nv_build` to + `anthropic`). Different models filter differently; spot-check the + five in-tree skills before merging the provider swap. - We observe a real-world skill that lands in an obviously wrong bucket (false positive or false negative). Open a tracking issue, link it from this doc, and adjust with evidence in the next PR. From c10bf521be005930f60a53acc870f347f9b12d6b Mon Sep 17 00:00:00 2001 From: DevCats Date: Mon, 22 Jun 2026 14:54:39 -0500 Subject: [PATCH 03/13] docs(README.md): document LLM mode and the one-time secret setup Update step 3 of the architecture summary to reflect that the scheduled scan now runs SkillSpector with the LLM semantic pass on by default. Add a new "One-time setup on the repo" section that lists the three repo-level configurations needed for a useful scan, including the new LLM credential secret. Mirror the LLM secret note into "Forking for your own catalogue". This commit was prepared with help from Coder Agents. --- README.md | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index f662a82..0aacfe4 100644 --- a/README.md +++ b/README.md @@ -8,8 +8,10 @@ Every 6 hours, the scheduled workflow in this repo: 1. Enumerates every skill in `coder/registry` (both the in-tree `.agents/skills/` format and the future external-sources format). 2. Shallow-clones each source repo. -3. Runs [NVIDIA SkillSpector](https://github.com/NVIDIA/SkillSpector) in - `--no-llm` static mode over the upstream content. +3. Runs [NVIDIA SkillSpector](https://github.com/NVIDIA/SkillSpector) over + the upstream content. The scheduled scan uses LLM semantic analysis + when the credential secret is configured, and falls back to + `--no-llm` static-only mode otherwise. 4. Builds a per-skill verdict (`clean`, `suspicious`, `malicious`, `unknown`) from `risk_score` plus the thresholds in `config.yaml`. 5. Builds the React SPA in `site/` and ships it together with @@ -60,6 +62,26 @@ Vite's dev proxy (see `site/vite.config.ts`) forwards `latest.json`, app sees real scanner output without CORS shenanigans. SPA routes such as `/skills/coder/setup` stay client-side. +## One-time setup on the repo + +Three things have to be configured once on the GitHub repo before the +scheduled scan publishes a useful result: + +1. **Settings > Pages**: set source to "GitHub Actions". The + `publish-pages` job in `scan.yaml` will fail until this is set. +2. **Settings > Actions**: workflow permissions "Read and write" so + `publish-release` can create the rolling `latest` release. +3. **Settings > Secrets and variables > Actions**: add the LLM + credential matching the provider in `config.yaml`'s + `scanners.skillspector.llm.provider`. For the default `nv_build` + provider this is `NVIDIA_INFERENCE_KEY` (sign up free at + [build.nvidia.com](https://build.nvidia.com)). Without the secret + the scan still runs, but SkillSpector falls back to + `--no-llm` static-only mode and precision drops from roughly 87% + to roughly 70%. See `docs/CALIBRATION.md` for the precision + discussion. The optional `SLACK_WEBHOOK_URL` secret enables the + `notify-slack-on-failure` job; without it that job is a no-op. + ## Repo layout ```text @@ -97,7 +119,10 @@ This scanner is data-driven. To run it against a different registry: "GitHub Actions"). 4. Set Actions workflow permissions to "Read and write" so the publish-release job can create releases. -5. Enable Actions. +5. Add the LLM credential secret matching your chosen provider + (see "One-time setup on the repo" above). Optional; static-only + mode works without it. +6. Enable Actions. No source changes required for catalogue changes. @@ -115,7 +140,8 @@ SkillSpector's `risk_score` (0-100) is the only input. The thresholds are aligned to SkillSpector's own `HIGH` and `CRITICAL` bands; [`docs/CALIBRATION.md`](./docs/CALIBRATION.md) walks through the evidence (SkillSpector source, the ClawHub paper, our in-tree -catalogue) behind the chosen numbers. +catalogue) behind the chosen numbers, and the LLM-on-vs-off precision +discussion behind running the semantic pass on every scheduled scan. The architecture keeps room for additional scanners (gitleaks, Semgrep, VirusTotal Premium, etc.); adding one is a new module under `scanner/`, From 8c57f9a637bec637cd63766f5402dedafba1bb01 Mon Sep 17 00:00:00 2001 From: DevCats Date: Mon, 22 Jun 2026 17:23:24 -0500 Subject: [PATCH 04/13] feat(config.yaml): switch LLM provider to Anthropic Sonnet 4.6 Swap the default LLM provider from nv_build (free NVIDIA Build) to anthropic with model pinned to claude-sonnet-4-6. Rationale: - Removes the second-vendor signup. The Coder org already has an Anthropic billing relationship, so the credential is one secret away from working. - Sonnet 4.6 is roughly 5x cheaper than the anthropic default (Opus 4.6) and is well matched to SkillSpector's LLM pass, which is finding-by-finding intent classification rather than long-form reasoning. Cost ballpark for 5 skills x 4 scans/day is small. - The other provider options (anthropic_proxy via Vertex, openai via any OpenAI-compatible gateway, nv_build) stay documented in the config comments and are still a one-line swap. This commit was prepared with help from Coder Agents. --- config.yaml | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/config.yaml b/config.yaml index 31be704..9cc2328 100644 --- a/config.yaml +++ b/config.yaml @@ -59,21 +59,23 @@ scanners: # Provider options and the env var SkillSpector consumes: # # provider env var(s) - # nv_build NVIDIA_INFERENCE_KEY (free; build.nvidia.com) - # openai OPENAI_API_KEY (+ OPENAI_BASE_URL for AI gateways) - # anthropic ANTHROPIC_API_KEY + # anthropic ANTHROPIC_API_KEY (api.anthropic.com) # anthropic_proxy ANTHROPIC_PROXY_API_KEY + ANTHROPIC_PROXY_ENDPOINT_URL + # openai OPENAI_API_KEY (+ OPENAI_BASE_URL for AI gateways) + # nv_build NVIDIA_INFERENCE_KEY (free; build.nvidia.com) # # Changing provider also requires updating the env block in # .github/workflows/scan.yaml so the matching secret is wired in, # and adding the secret under Settings > Secrets and variables > # Actions. llm: - provider: nv_build - # Empty model uses the provider's bundled default. Override here - # to pin a specific revision (e.g. "claude-opus-4-6" for - # provider=anthropic). - model: "" + provider: anthropic + # SkillSpector's bundled default for the anthropic provider is + # claude-opus-4-6. Sonnet 4.6 is roughly 5x cheaper than Opus and + # is well-suited for the finding-classification task the LLM pass + # actually does, so it is the better cost/quality choice for + # periodic scanning. Override here to pin a different revision. + model: "claude-sonnet-4-6" # Per-skill verdict policy. v1 has one input (SkillSpector risk_score). # When more scanners join the pipeline we add new threshold fields here From 3333b8117369b441b8697350984b8cd036a915a9 Mon Sep 17 00:00:00 2001 From: DevCats Date: Mon, 22 Jun 2026 17:24:12 -0500 Subject: [PATCH 05/13] docs(README.md): point setup at Anthropic provider, not nv_build Follow-up to the provider swap. The one-time-setup section now points at console.anthropic.com and ANTHROPIC_API_KEY instead of build.nvidia.com / NVIDIA_INFERENCE_KEY. This commit was prepared with help from Coder Agents. --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 0aacfe4..f486179 100644 --- a/README.md +++ b/README.md @@ -73,10 +73,10 @@ scheduled scan publishes a useful result: `publish-release` can create the rolling `latest` release. 3. **Settings > Secrets and variables > Actions**: add the LLM credential matching the provider in `config.yaml`'s - `scanners.skillspector.llm.provider`. For the default `nv_build` - provider this is `NVIDIA_INFERENCE_KEY` (sign up free at - [build.nvidia.com](https://build.nvidia.com)). Without the secret - the scan still runs, but SkillSpector falls back to + `scanners.skillspector.llm.provider`. For the default `anthropic` + provider this is `ANTHROPIC_API_KEY` (from + [console.anthropic.com](https://console.anthropic.com)). Without + the secret the scan still runs, but SkillSpector falls back to `--no-llm` static-only mode and precision drops from roughly 87% to roughly 70%. See `docs/CALIBRATION.md` for the precision discussion. The optional `SLACK_WEBHOOK_URL` secret enables the From 28c05fe24f945667e07f251dc4fae0e24ec9edac Mon Sep 17 00:00:00 2001 From: DevCats Date: Tue, 23 Jun 2026 15:30:43 -0500 Subject: [PATCH 06/13] fix(config.yaml): reframe LLM block as contract, swap to openai+aibridge Addresses copilot-pull-request-reviewer review on config.yaml line 46 and line 57: the previous comments described workflow behavior as implemented ("the workflow appends --no-llm dynamically") when in fact .github/workflows/scan.yaml still hardcodes --no-llm in this PR. The new wording describes the contract between this file and scan.yaml and explicitly notes that the matching workflow edit is committed separately because the Coder Agents GitHub App lacks `workflows: write`. Also swaps the provider from anthropic/claude-sonnet-4-6 to openai/gpt-4.1-mini. The anthropic provider hardcodes api.anthropic.com and ignores ANTHROPIC_BASE_URL, so it cannot route through Coder's aibridge. The openai provider does, and gpt-4.1-mini was empirically validated against the five in-tree skills (results in CALIBRATION.md). This commit was prepared with help from Coder Agents. --- config.yaml | 64 +++++++++++++++++++++++++++++++---------------------- 1 file changed, 37 insertions(+), 27 deletions(-) diff --git a/config.yaml b/config.yaml index 9cc2328..7ac5268 100644 --- a/config.yaml +++ b/config.yaml @@ -39,43 +39,53 @@ scanners: # so a bumper bot lives outside the loop until the upstream # publishes to PyPI and the pin can move into pyproject.toml. pin: "skillspector @ git+https://github.com/NVIDIA/SkillSpector.git@2eb844780ab163f01468ecf142c40a2ec0fcaec0" - # Extra CLI flags passed to every SkillSpector invocation. Empty by - # default; the scan workflow appends --no-llm dynamically when the - # LLM credential secret is not set (see llm: block below). CI runs - # do not invoke SkillSpector live. + # Extra CLI flags passed to every SkillSpector invocation. Left + # empty so .github/workflows/scan.yaml can drive --no-llm + # dynamically based on whether the LLM credential secret is set + # (see contract under the llm: block below). flags: [] - # SkillSpector ships a two-stage analyser: fast static rules followed - # by an optional LLM semantic pass. The LLM pass lifts precision - # from roughly 70% to roughly 87% per upstream docs by filtering - # context-aware false positives, classifying intent on prompt - # injection patterns, and producing human-readable explanations. + # SkillSpector ships a two-stage analyser: fast static rules + # followed by an optional LLM semantic pass. Measured on the five + # in-tree skills, LLM mode dropped catalogue-wide findings from + # 25 to 2 and moved coder/setup from malicious to clean. See + # docs/CALIBRATION.md for the per-skill numbers and methodology. # - # The scheduled scan reads the credential matching the provider - # below from a repository secret. When the secret is configured, - # LLM mode is on. When the secret is missing, the workflow falls - # back to --no-llm automatically so a fresh fork is never broken - # by an unset secret. + # Contract with .github/workflows/scan.yaml: the workflow reads + # the credential matching the provider below from a repository + # secret. When the secret is set, LLM mode runs. When the secret + # is missing, the workflow appends --no-llm so a fresh fork is + # never broken by an unset secret. # - # Provider options and the env var SkillSpector consumes: + # NOTE: scan.yaml in this branch still hardcodes --no-llm. The + # matching workflow edit is in this PR's description but is + # committed separately because the Coder Agents GitHub App on + # this repo currently lacks the `workflows: write` scope. After + # both land, the config and the workflow drive LLM mode together. # - # provider env var(s) - # anthropic ANTHROPIC_API_KEY (api.anthropic.com) - # anthropic_proxy ANTHROPIC_PROXY_API_KEY + ANTHROPIC_PROXY_ENDPOINT_URL - # openai OPENAI_API_KEY (+ OPENAI_BASE_URL for AI gateways) - # nv_build NVIDIA_INFERENCE_KEY (free; build.nvidia.com) + # Provider options: + # provider env var(s) endpoint + # openai OPENAI_API_KEY + OPENAI_BASE_URL any OpenAI-compatible URL + # anthropic ANTHROPIC_API_KEY api.anthropic.com (hardcoded; + # ignores ANTHROPIC_BASE_URL) + # nv_build NVIDIA_INFERENCE_KEY build.nvidia.com (free) # # Changing provider also requires updating the env block in # .github/workflows/scan.yaml so the matching secret is wired in, # and adding the secret under Settings > Secrets and variables > # Actions. llm: - provider: anthropic - # SkillSpector's bundled default for the anthropic provider is - # claude-opus-4-6. Sonnet 4.6 is roughly 5x cheaper than Opus and - # is well-suited for the finding-classification task the LLM pass - # actually does, so it is the better cost/quality choice for - # periodic scanning. Override here to pin a different revision. - model: "claude-sonnet-4-6" + # Routes through Coder's AI Gateway (aibridge). OPENAI_BASE_URL + # is set as a repo variable to: + # https://dev.coder.com/api/v2/aibridge/openai/v1 + # The openai-compatible path is used because SkillSpector's + # `anthropic` provider hardcodes api.anthropic.com and ignores + # ANTHROPIC_BASE_URL, so it cannot be steered at aibridge. + provider: openai + # gpt-4.1-mini is the cost/quality sweet spot for SkillSpector's + # finding-classification work and is the model the calibration + # in docs/CALIBRATION.md was measured against. Override to pin + # a different revision (e.g. gpt-5.4-mini, gpt-4.1). + model: "gpt-4.1-mini" # Per-skill verdict policy. v1 has one input (SkillSpector risk_score). # When more scanners join the pipeline we add new threshold fields here From 6bbb6bc27c4df93cc50c8a4b0a9bac4e0f9dee84 Mon Sep 17 00:00:00 2001 From: DevCats Date: Tue, 23 Jun 2026 15:31:52 -0500 Subject: [PATCH 07/13] fix(docs/CALIBRATION.md): real measured numbers + workflow-gap note Addresses copilot-pull-request-reviewer review on docs/CALIBRATION.md line 121: the LLM section described auto-enablement based on the credential secret, but scan.yaml in this branch still hardcodes --no-llm and that contract is not in effect yet. This pass: - Adds a measured table showing the five-skill before/after under LLM mode (was upstream's hand-wavy 70%-to-87% precision estimate). Catalogue-wide findings: 25 to 2; coder/setup: malicious to clean. - Adds an explicit "workflow gap" callout explaining that scan.yaml still hardcodes --no-llm in this branch and pointing readers at the PR description for the matching diff. - Documents the provider choice in plain language: anthropic provider cannot be steered at aibridge because it hardcodes api.anthropic.com and ignores ANTHROPIC_BASE_URL; openai provider works and gpt-4.1-mini is the cost/quality sweet spot. - Drops the previous "bringing coder/setup below suspicious requires the permissions-manifest layer" framing. With LLM mode on, coder/setup is already clean. This commit was prepared with help from Coder Agents. --- docs/CALIBRATION.md | 112 ++++++++++++++++++++++++++++---------------- 1 file changed, 72 insertions(+), 40 deletions(-) diff --git a/docs/CALIBRATION.md b/docs/CALIBRATION.md index 5091163..bb362c9 100644 --- a/docs/CALIBRATION.md +++ b/docs/CALIBRATION.md @@ -64,18 +64,18 @@ The current `coder/registry` in-tree catalogue contains five skills: `coder/coder-modules`, `coder/coder-templates`, `coder/modules`, `coder/templates`, and `coder/setup`. Under the chosen thresholds: -| Skill | SkillSpector score | Verdict | -|------------------------|-------------------:|-------------| -| `coder/coder-modules` | 0 | `clean` | -| `coder/coder-templates`| 0 | `clean` | -| `coder/modules` | 0 | `clean` | -| `coder/templates` | 10 | `clean` | -| `coder/setup` | 100 | `malicious` | +| Skill | static score | LLM-mode score | static verdict | LLM-mode verdict | +|------------------------|-------------:|---------------:|----------------|------------------| +| `coder/coder-modules` | 10 | 0 | `clean` | `clean` | +| `coder/coder-templates`| 10 | 0 | `clean` | `clean` | +| `coder/modules` | 0 | 0 | `clean` | `clean` | +| `coder/templates` | 0 | 0 | `clean` | `clean` | +| `coder/setup` | 100 | 26 | `malicious` | `clean` | The previous thresholds (40/75) produced the same outcome for these -five inputs. The change does not silence any signal that was firing -today; it raises the bar that future skills must clear before being -called out. +five inputs under static-only mode. The change does not silence any +signal that was firing today; it raises the bar that future skills +must clear before being called out. ## Threshold choices @@ -103,37 +103,68 @@ verdict: SkillSpector ships a two-stage analyser: fast static rules (the 64 patterns SkillSpector documents) followed by an optional LLM semantic -pass. Upstream's published precision numbers are: - -- `--no-llm` (static only): high recall, moderate precision (~70%). - False positives on context-sensitive patterns are common; for - example, EA2 ("autonomous decision making") fires on prose that - documents safeguards as well as prose that bypasses them. -- Default (LLM on): ~87% precision. The LLM pass reads each finding's - surrounding context, classifies intent, filters context-aware false - positives, and writes a human-readable explanation that ships in the - per-finding output. +pass. The LLM pass reads each finding's surrounding context, classifies +intent, filters context-aware false positives, and writes a +human-readable explanation that ships in the per-finding output. + +### Measured impact on the five in-tree skills + +Measured against `gpt-4.1-mini` through Coder's AI Gateway. Methodology: +ran `skillspector scan` twice on each upstream skill (once with +`--no-llm`, once with LLM mode on) and aggregated the per-skill +results. Total catalogue-wide findings dropped from 25 to 2: + +| Skill | findings (static) | findings (LLM) | Δ | +|------------------------|------------------:|---------------:|----------| +| `coder/coder-modules` | 1 | 0 | -1 | +| `coder/coder-templates`| 1 | 0 | -1 | +| `coder/modules` | 0 | 0 | 0 | +| `coder/setup` | 23 | 2 | -21 | +| `coder/templates` | 0 | 0 | 0 | +| **TOTAL** | **25** | **2** | **-23** | + +`coder/setup`'s verdict moves from `malicious` (100) to `clean` (26). +The LLM filtered all 23 static-only findings as context-aware false +positives (the EA2 hits on safeguard prose, the MP2 hits on PNG +assets, the SC2 hits on `curl coder.com/install.sh`, the PE3 hits on +the skill's own scratch files, etc.) and surfaced 2 new MEDIUM +findings (`SQP-2`) the static pass missed: the GitHub device-flow +scripts write the OAuth token and session config to disk without a +user-visible notification. Those 2 findings are real and minor; the +cleanest fix is a one-line `echo` before each write in the upstream +skill repo rather than any change here. + +### Provider choice and the workflow gap The scheduled scan runs LLM mode when the workflow's chosen credential -secret (`NVIDIA_INFERENCE_KEY` for the default `nv_build` provider) is -configured. The fallback to `--no-llm` is automatic when the secret is -missing, so an unset secret on a fresh fork degrades the scan rather -than breaking it. - -The LLM pass does not affect the threshold math: SkillSpector's +secret is configured. The fallback to `--no-llm` is automatic when the +secret is missing, so an unset secret on a fresh fork degrades the +scan rather than breaking it. + +**Important caveat for this PR**: `.github/workflows/scan.yaml` in the +current branch still hardcodes `--no-llm`. The matching workflow edit +is in the PR description but is committed separately because the Coder +Agents GitHub App on this repo currently lacks the `workflows: write` +scope. The contract documented here only takes effect once that edit +lands (or is pasted by a human with workflow write access). + +Provider is `openai` against Coder's AI Gateway endpoint +(`OPENAI_BASE_URL=https://dev.coder.com/api/v2/aibridge/openai/v1`) +with model `gpt-4.1-mini`. SkillSpector's `anthropic` provider was +tried first because it would map more directly to claude-sonnet-class +models, but its `provider.py` hardcodes `https://api.anthropic.com/v1/` +and ignores `ANTHROPIC_BASE_URL`, so it cannot be steered at aibridge. +The `openai` provider does respect `OPENAI_BASE_URL`, and aibridge +exposes `gpt-4.1-mini` plus a long list of other OpenAI-class models. + +### How the LLM pass interacts with the verdict math + +The LLM pass does not affect the threshold math. SkillSpector's `risk_score` is still a 0-100 weighted sum of rule hits, and the 51/81 cutoffs above still map directly to `HIGH` and `CRITICAL` bands. -It does affect which findings reach the verdict: false positives that -the LLM filters out no longer contribute to the score. Expect verdicts -to move down (or stay the same) when LLM mode flips on, not up. - -For the five existing in-tree skills, the static-only scan placed -`coder/setup` at 100 / `malicious`. With LLM mode on we expect the -findings list to shrink (the EA2 prose hits and the asset-path MP2 -hits should be filtered) but the score will still be high. Reducing -`coder/setup`'s verdict below `suspicious` requires the upcoming -permissions-manifest layer (Phase 3 of the v3 plan), not the LLM pass -alone. +What changes is which findings reach the verdict: false positives the +LLM filters out no longer contribute to the score. Verdicts move down +(or stay the same) when LLM mode flips on, not up. ## What we did not change (and why) @@ -163,9 +194,10 @@ Re-run this analysis when any of: that shifts where its bands sit. The pinned commit in `config.yaml` protects us from drifting silently; a deliberate bump should walk through this doc. -- The LLM provider changes (e.g., moving from `nv_build` to - `anthropic`). Different models filter differently; spot-check the - five in-tree skills before merging the provider swap. +- The LLM model or provider changes (e.g., moving from `gpt-4.1-mini` + to a Claude or Gemini model, or from aibridge to a direct provider + key). Different models filter differently; spot-check the five + in-tree skills before merging the provider swap. - We observe a real-world skill that lands in an obviously wrong bucket (false positive or false negative). Open a tracking issue, link it from this doc, and adjust with evidence in the next PR. From b301daaf955f90ef9bec7f9236d31a697c318064 Mon Sep 17 00:00:00 2001 From: DevCats Date: Tue, 23 Jun 2026 15:32:52 -0500 Subject: [PATCH 08/13] fix(README.md): clarify the workflow-file dependency, point setup at aibridge Addresses copilot-pull-request-reviewer reviews on README.md (architecture step 3 at line 14, the one-time setup block, the graceful-fallback paragraph, and the forking step 5 at line 125): the text described workflow behavior that is not implemented in this branch because .github/workflows/scan.yaml still hardcodes --no-llm. Fixes: - Add a note block right after the architecture summary spelling out that step 3's LLM behavior requires the matching scan.yaml edit, and pointing readers at the PR description for the diff. - Update step 3 of the one-time-setup section to set the OpenAI provider's secret and add OPENAI_BASE_URL as a variable. Adds a second note block immediately after the setup list flagging the workflow-file dependency again at point-of-use. - Update forking instructions step 5 to point at the new setup section and call out that confirming scan.yaml exports the secret is required for LLM mode. - Swap the API-key reference from console.anthropic.com / ANTHROPIC_API_KEY to a Coder AI Gateway token / OPENAI_API_KEY. The anthropic provider in SkillSpector hardcodes api.anthropic.com and ignores ANTHROPIC_BASE_URL, so it cannot be steered at aibridge; the openai provider works. This commit was prepared with help from Coder Agents. --- README.md | 57 +++++++++++++++++++++++++++++++++++++------------------ 1 file changed, 39 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index f486179..c271fa7 100644 --- a/README.md +++ b/README.md @@ -9,9 +9,9 @@ Every 6 hours, the scheduled workflow in this repo: `.agents/skills/` format and the future external-sources format). 2. Shallow-clones each source repo. 3. Runs [NVIDIA SkillSpector](https://github.com/NVIDIA/SkillSpector) over - the upstream content. The scheduled scan uses LLM semantic analysis - when the credential secret is configured, and falls back to - `--no-llm` static-only mode otherwise. + the upstream content. The scheduled scan runs SkillSpector's LLM + semantic pass when the workflow's LLM credential secret is + configured, and falls back to `--no-llm` static-only mode otherwise. 4. Builds a per-skill verdict (`clean`, `suspicious`, `malicious`, `unknown`) from `risk_score` plus the thresholds in `config.yaml`. 5. Builds the React SPA in `site/` and ships it together with @@ -27,6 +27,13 @@ The registry site reads the public report through a small proxy endpoint in `coder/registry-server` (separate PR) and shows a per-skill badge. The registry's deploys are not gated on the scan result. +> **Note on the LLM semantic pass.** Step 3's conditional-LLM behavior +> requires the matching edit in `.github/workflows/scan.yaml`. That edit +> is part of the same change that introduces this section in the README; +> see the PR description for the diff. The current `scan.yaml` on this +> branch still hardcodes `--no-llm`, so adding the secret alone has no +> effect until the workflow edit also lands. + ## Reading the latest report Stable URLs, no auth required: @@ -64,24 +71,36 @@ as `/skills/coder/setup` stay client-side. ## One-time setup on the repo -Three things have to be configured once on the GitHub repo before the +Four things have to be configured once on the GitHub repo before the scheduled scan publishes a useful result: 1. **Settings > Pages**: set source to "GitHub Actions". The `publish-pages` job in `scan.yaml` will fail until this is set. 2. **Settings > Actions**: workflow permissions "Read and write" so `publish-release` can create the rolling `latest` release. -3. **Settings > Secrets and variables > Actions**: add the LLM - credential matching the provider in `config.yaml`'s - `scanners.skillspector.llm.provider`. For the default `anthropic` - provider this is `ANTHROPIC_API_KEY` (from - [console.anthropic.com](https://console.anthropic.com)). Without - the secret the scan still runs, but SkillSpector falls back to - `--no-llm` static-only mode and precision drops from roughly 87% - to roughly 70%. See `docs/CALIBRATION.md` for the precision - discussion. The optional `SLACK_WEBHOOK_URL` secret enables the +3. **Settings > Secrets and variables > Actions > Variables**: add + `OPENAI_BASE_URL` with the value + `https://dev.coder.com/api/v2/aibridge/openai/v1`. This is a + variable (not a secret) because it is not sensitive; only the API + key is. +4. **Settings > Secrets and variables > Actions > Secrets**: add the + LLM credential matching the provider in `config.yaml`'s + `scanners.skillspector.llm.provider`. For the default `openai` + provider this is `OPENAI_API_KEY` set to a Coder AI Gateway token. + Without the secret, the scan still runs but SkillSpector falls + back to `--no-llm` static-only mode and precision drops. See + `docs/CALIBRATION.md` for the measured before/after numbers. The + optional `SLACK_WEBHOOK_URL` secret enables the `notify-slack-on-failure` job; without it that job is a no-op. +> **Workflow file note**: enabling LLM mode also requires the matching +> edit in `.github/workflows/scan.yaml` (it must export the secret +> into the SkillSpector step and conditionally append `--no-llm`). +> That edit is part of the same change that introduces this section +> and is documented in the PR description, but is committed +> separately because the Coder Agents GitHub App on this repo +> currently lacks the `workflows: write` scope. + ## Repo layout ```text @@ -119,9 +138,11 @@ This scanner is data-driven. To run it against a different registry: "GitHub Actions"). 4. Set Actions workflow permissions to "Read and write" so the publish-release job can create releases. -5. Add the LLM credential secret matching your chosen provider - (see "One-time setup on the repo" above). Optional; static-only - mode works without it. +5. To enable the LLM semantic pass, add the credential secret and the + matching `OPENAI_BASE_URL` variable per "One-time setup on the + repo" above, AND confirm `.github/workflows/scan.yaml` exports the + secret into the SkillSpector step. Static-only mode (without the + secret) is the default and works out of the box. 6. Enable Actions. No source changes required for catalogue changes. @@ -140,8 +161,8 @@ SkillSpector's `risk_score` (0-100) is the only input. The thresholds are aligned to SkillSpector's own `HIGH` and `CRITICAL` bands; [`docs/CALIBRATION.md`](./docs/CALIBRATION.md) walks through the evidence (SkillSpector source, the ClawHub paper, our in-tree -catalogue) behind the chosen numbers, and the LLM-on-vs-off precision -discussion behind running the semantic pass on every scheduled scan. +catalogue) behind the chosen numbers and the measured LLM-on-vs-off +impact on the five in-tree skills. The architecture keeps room for additional scanners (gitleaks, Semgrep, VirusTotal Premium, etc.); adding one is a new module under `scanner/`, From 0fea10e604b270140cdbe0e6c4c3c0556d5242e3 Mon Sep 17 00:00:00 2001 From: DevCats Date: Tue, 23 Jun 2026 21:07:42 +0000 Subject: [PATCH 09/13] fix(config.yaml): swap LLM provider to Anthropic direct, not aibridge SkillSpector's anthropic provider hardcodes api.anthropic.com and ignores ANTHROPIC_BASE_URL, and aibridge does not mount /v1/chat/completions on its /anthropic path (only the native /v1/messages shape). Routing SkillSpector at Claude through aibridge is therefore not possible today without either patching SkillSpector or adding a new route to aibridge. Use the Anthropic API directly instead. This trades the aibridge billing-line consolidation for a Claude-class model; the choice and the trade-off are documented in docs/CALIBRATION.md. --- config.yaml | 35 +++++++++++++++++++++-------------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/config.yaml b/config.yaml index 7ac5268..f1b135f 100644 --- a/config.yaml +++ b/config.yaml @@ -64,9 +64,8 @@ scanners: # # Provider options: # provider env var(s) endpoint + # anthropic ANTHROPIC_API_KEY api.anthropic.com # openai OPENAI_API_KEY + OPENAI_BASE_URL any OpenAI-compatible URL - # anthropic ANTHROPIC_API_KEY api.anthropic.com (hardcoded; - # ignores ANTHROPIC_BASE_URL) # nv_build NVIDIA_INFERENCE_KEY build.nvidia.com (free) # # Changing provider also requires updating the env block in @@ -74,18 +73,26 @@ scanners: # and adding the secret under Settings > Secrets and variables > # Actions. llm: - # Routes through Coder's AI Gateway (aibridge). OPENAI_BASE_URL - # is set as a repo variable to: - # https://dev.coder.com/api/v2/aibridge/openai/v1 - # The openai-compatible path is used because SkillSpector's - # `anthropic` provider hardcodes api.anthropic.com and ignores - # ANTHROPIC_BASE_URL, so it cannot be steered at aibridge. - provider: openai - # gpt-4.1-mini is the cost/quality sweet spot for SkillSpector's - # finding-classification work and is the model the calibration - # in docs/CALIBRATION.md was measured against. Override to pin - # a different revision (e.g. gpt-5.4-mini, gpt-4.1). - model: "gpt-4.1-mini" + # Hits api.anthropic.com directly with an Anthropic API key (not + # routed through Coder's AI Gateway). aibridge was the original + # plan, but SkillSpector pipes every provider through + # langchain_openai.ChatOpenAI which hits `/v1/chat/completions`, + # and aibridge only mounts that path under `/openai`, not + # `/anthropic`. The `anthropic` provider in SkillSpector also + # hardcodes `https://api.anthropic.com/v1/` and ignores + # ANTHROPIC_BASE_URL, so aibridge cannot be steered into the + # Claude path today. This means the Anthropic API key is on a + # separate billing line from Coder usage. + provider: anthropic + # claude-sonnet-4-5 is the cost/quality sweet spot for + # SkillSpector's finding-classification work. SkillSpector's + # bundled default for the anthropic provider is + # `claude-opus-4-6`, which is ~5x the per-token cost; the + # finding-classification task does not need Opus-class + # reasoning. Bump to a newer Sonnet revision (or to Opus) by + # editing this string; SkillSpector passes it directly to the + # Anthropic API. + model: "claude-sonnet-4-5-20250929" # Per-skill verdict policy. v1 has one input (SkillSpector risk_score). # When more scanners join the pipeline we add new threshold fields here From f10bfade9d14ad17699bda0fc0a1d7e2b06d077e Mon Sep 17 00:00:00 2001 From: DevCats Date: Tue, 23 Jun 2026 21:07:42 +0000 Subject: [PATCH 10/13] docs(README.md): point setup at ANTHROPIC_API_KEY, drop OPENAI_BASE_URL var Three-step one-time setup now: Pages, workflow permissions, and a single secret (ANTHROPIC_API_KEY from console.anthropic.com). The OPENAI_BASE_URL variable is no longer used since the provider is anthropic against api.anthropic.com directly. The workflow-file note spells out the matching workflow env block (SKILLSPECTOR_PROVIDER, SKILLSPECTOR_MODEL, ANTHROPIC_API_KEY) that has to land in scan.yaml. --- README.md | 40 ++++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index c271fa7..8a7cefc 100644 --- a/README.md +++ b/README.md @@ -71,22 +71,20 @@ as `/skills/coder/setup` stay client-side. ## One-time setup on the repo -Four things have to be configured once on the GitHub repo before the +Three things have to be configured once on the GitHub repo before the scheduled scan publishes a useful result: 1. **Settings > Pages**: set source to "GitHub Actions". The `publish-pages` job in `scan.yaml` will fail until this is set. 2. **Settings > Actions**: workflow permissions "Read and write" so `publish-release` can create the rolling `latest` release. -3. **Settings > Secrets and variables > Actions > Variables**: add - `OPENAI_BASE_URL` with the value - `https://dev.coder.com/api/v2/aibridge/openai/v1`. This is a - variable (not a secret) because it is not sensitive; only the API - key is. -4. **Settings > Secrets and variables > Actions > Secrets**: add the +3. **Settings > Secrets and variables > Actions > Secrets**: add the LLM credential matching the provider in `config.yaml`'s - `scanners.skillspector.llm.provider`. For the default `openai` - provider this is `OPENAI_API_KEY` set to a Coder AI Gateway token. + `scanners.skillspector.llm.provider`. For the default `anthropic` + provider this is `ANTHROPIC_API_KEY` (from + [console.anthropic.com](https://console.anthropic.com); this is a + separate billing line from Coder usage because SkillSpector cannot + be routed through aibridge today, see `docs/CALIBRATION.md`). Without the secret, the scan still runs but SkillSpector falls back to `--no-llm` static-only mode and precision drops. See `docs/CALIBRATION.md` for the measured before/after numbers. The @@ -94,12 +92,14 @@ scheduled scan publishes a useful result: `notify-slack-on-failure` job; without it that job is a no-op. > **Workflow file note**: enabling LLM mode also requires the matching -> edit in `.github/workflows/scan.yaml` (it must export the secret -> into the SkillSpector step and conditionally append `--no-llm`). -> That edit is part of the same change that introduces this section -> and is documented in the PR description, but is committed -> separately because the Coder Agents GitHub App on this repo -> currently lacks the `workflows: write` scope. +> edit in `.github/workflows/scan.yaml` (it must export +> `ANTHROPIC_API_KEY` into the SkillSpector step, set +> `SKILLSPECTOR_PROVIDER=anthropic` and `SKILLSPECTOR_MODEL`, and +> conditionally append `--no-llm` when the secret is missing). That +> edit is part of the same change that introduces this section and +> is documented in the PR description, but is committed separately +> because the Coder Agents GitHub App on this repo currently lacks +> the `workflows: write` scope. ## Repo layout @@ -138,11 +138,11 @@ This scanner is data-driven. To run it against a different registry: "GitHub Actions"). 4. Set Actions workflow permissions to "Read and write" so the publish-release job can create releases. -5. To enable the LLM semantic pass, add the credential secret and the - matching `OPENAI_BASE_URL` variable per "One-time setup on the - repo" above, AND confirm `.github/workflows/scan.yaml` exports the - secret into the SkillSpector step. Static-only mode (without the - secret) is the default and works out of the box. +5. To enable the LLM semantic pass, add the credential secret per + "One-time setup on the repo" above, AND confirm + `.github/workflows/scan.yaml` exports the secret into the + SkillSpector step. Static-only mode (without the secret) is the + default and works out of the box. 6. Enable Actions. No source changes required for catalogue changes. From d6c9c5dca11d8e57f6ff3f586660bb6a90a230e8 Mon Sep 17 00:00:00 2001 From: DevCats Date: Tue, 23 Jun 2026 21:07:42 +0000 Subject: [PATCH 11/13] docs(CALIBRATION.md): record Anthropic-direct decision and model-swap caveat The measured 25 -> 2 false-positive reduction was captured against gpt-4.1-mini through aibridge during development; production hits claude-sonnet-4-5 via the Anthropic API directly. Verdict-band outcomes are robust to the model swap because every non-coder/setup in-tree skill scores well below the 51 suspicious cutoff even without LLM filtering, but the per-finding counts may shift one or two either way. Recalibration follow-up planned once production data lands. Also captures the four reasons aibridge cannot route SkillSpector at Claude today. --- docs/CALIBRATION.md | 62 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 48 insertions(+), 14 deletions(-) diff --git a/docs/CALIBRATION.md b/docs/CALIBRATION.md index bb362c9..4e2879c 100644 --- a/docs/CALIBRATION.md +++ b/docs/CALIBRATION.md @@ -109,8 +109,9 @@ human-readable explanation that ships in the per-finding output. ### Measured impact on the five in-tree skills -Measured against `gpt-4.1-mini` through Coder's AI Gateway. Methodology: -ran `skillspector scan` twice on each upstream skill (once with +Measured against `gpt-4.1-mini` through Coder's AI Gateway during +development, before the provider swap below. Methodology: ran +`skillspector scan` twice on each upstream skill (once with `--no-llm`, once with LLM mode on) and aggregated the per-skill results. Total catalogue-wide findings dropped from 25 to 2: @@ -134,6 +135,19 @@ user-visible notification. Those 2 findings are real and minor; the cleanest fix is a one-line `echo` before each write in the upstream skill repo rather than any change here. +**Model swap caveat**: production runs against `claude-sonnet-4-5` +via the Anthropic API (see "Provider choice" below), not against +`gpt-4.1-mini`. The 25 → 2 delta above measures SkillSpector's LLM +semantic pass *as a capability*; absolute counts may shift one or two +either way under Claude because the two models filter false positives +slightly differently. The verdict-band outcomes (`coder/setup` flips +malicious → clean, every other in-tree skill stays clean) are robust +to that drift: every static finding on the four other skills is well +below the `suspicious_risk_score: 51` cutoff to begin with, so even a +100% no-filter LLM still leaves them clean. Recalibration against +Claude is a 30-minute follow-up PR once the secret is wired in and +the first production scan lands; this doc gets the real numbers then. + ### Provider choice and the workflow gap The scheduled scan runs LLM mode when the workflow's chosen credential @@ -148,14 +162,33 @@ Agents GitHub App on this repo currently lacks the `workflows: write` scope. The contract documented here only takes effect once that edit lands (or is pasted by a human with workflow write access). -Provider is `openai` against Coder's AI Gateway endpoint -(`OPENAI_BASE_URL=https://dev.coder.com/api/v2/aibridge/openai/v1`) -with model `gpt-4.1-mini`. SkillSpector's `anthropic` provider was -tried first because it would map more directly to claude-sonnet-class -models, but its `provider.py` hardcodes `https://api.anthropic.com/v1/` -and ignores `ANTHROPIC_BASE_URL`, so it cannot be steered at aibridge. -The `openai` provider does respect `OPENAI_BASE_URL`, and aibridge -exposes `gpt-4.1-mini` plus a long list of other OpenAI-class models. +Provider is `anthropic` against `api.anthropic.com` directly, model +`claude-sonnet-4-5-20250929`. The Anthropic API key is on a separate +billing line from Coder usage because SkillSpector cannot be routed +through Coder's AI Gateway today: + +- aibridge does proxy Claude under its `/anthropic` path, but only in + Anthropic's native `/v1/messages` shape. +- SkillSpector pipes every provider through + `langchain_openai.ChatOpenAI`, which speaks OpenAI's + `/v1/chat/completions` shape. +- aibridge does not mount `/v1/chat/completions` on its `/anthropic` + path (verified: `route not supported`). +- SkillSpector's `anthropic` provider also hardcodes + `https://api.anthropic.com/v1/` in `providers/anthropic/provider.py` + and ignores `ANTHROPIC_BASE_URL`, so even if aibridge did expose the + OpenAI-compat route on its Anthropic path, an env-only swap would + not steer SkillSpector at it. + +Using `openai` against aibridge with `gpt-4.1-mini` is a viable +alternative (and is what the calibration table above was measured +against). The trade-off is real: aibridge routing keeps inference +spend on Coder's existing billing line and avoids a second vendor, +but commits the scanner to whichever OpenAI-class model aibridge +exposes rather than Claude. If aibridge later adds either a Claude +OpenAI-compat route on `/anthropic` or a native-Anthropic +integration into SkillSpector, the provider line in `config.yaml` +flips back without any workflow change. ### How the LLM pass interacts with the verdict math @@ -194,10 +227,11 @@ Re-run this analysis when any of: that shifts where its bands sit. The pinned commit in `config.yaml` protects us from drifting silently; a deliberate bump should walk through this doc. -- The LLM model or provider changes (e.g., moving from `gpt-4.1-mini` - to a Claude or Gemini model, or from aibridge to a direct provider - key). Different models filter differently; spot-check the five - in-tree skills before merging the provider swap. +- The LLM model or provider changes (e.g., moving from + `claude-sonnet-4-5` to Opus or to a non-Anthropic provider). + Different models filter differently; spot-check the five in-tree + skills before merging the provider swap and refresh the table + above. - We observe a real-world skill that lands in an obviously wrong bucket (false positive or false negative). Open a tracking issue, link it from this doc, and adjust with evidence in the next PR. From 9f4080166423f831b4740497b45eee363758166f Mon Sep 17 00:00:00 2001 From: DevCats Date: Tue, 23 Jun 2026 21:14:19 +0000 Subject: [PATCH 12/13] feat(scanner): bump LLM model from claude-sonnet-4-5 to claude-sonnet-4-6 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit sonnet-4-6 is the bundled meta_analyzer-slot default in SkillSpector's anthropic provider — the slot whose docstring explicitly describes it as 'cheaper for the high-volume filter pass,' which is exactly the per-finding intent classification this scanner does. It also widens the context window from 200K (Sonnet 4.5) to 1M, reducing chunking for larger skills, and SkillSpector's bundled model_registry.yaml already knows its token limits so there is no runtime fallback warning. Bare alias used rather than a dated suffix: probing Anthropic's /v1/models endpoint on 2026-06-23 confirmed every dated suffix for the 4-6 series (e.g. claude-sonnet-4-6-20251015) returns 404, while the bare alias returns 200. The bare alias is the canonical ID Anthropic accepts. --- config.yaml | 24 +++++++++++++++--------- docs/CALIBRATION.md | 16 ++++++++-------- 2 files changed, 23 insertions(+), 17 deletions(-) diff --git a/config.yaml b/config.yaml index f1b135f..29805e5 100644 --- a/config.yaml +++ b/config.yaml @@ -84,15 +84,21 @@ scanners: # Claude path today. This means the Anthropic API key is on a # separate billing line from Coder usage. provider: anthropic - # claude-sonnet-4-5 is the cost/quality sweet spot for - # SkillSpector's finding-classification work. SkillSpector's - # bundled default for the anthropic provider is - # `claude-opus-4-6`, which is ~5x the per-token cost; the - # finding-classification task does not need Opus-class - # reasoning. Bump to a newer Sonnet revision (or to Opus) by - # editing this string; SkillSpector passes it directly to the - # Anthropic API. - model: "claude-sonnet-4-5-20250929" + # claude-sonnet-4-6 is SkillSpector's own pick for the + # finding-classification task: it is the bundled default for + # the anthropic provider's `meta_analyzer` slot, which the + # provider docstring describes as "cheaper for the high-volume + # filter pass." SkillSpector's `DEFAULT_MODEL` is + # `claude-opus-4-6`, used for the deeper analyzer slots + # (~5x the per-token cost); the per-finding intent + # classification this scanner relies on does not need + # Opus-class reasoning. Bare alias rather than a dated suffix + # (the dated suffixes for the 4-6 series are not accepted as + # of June 2026; the bare alias is the canonical ID). Bump + # this string to `claude-opus-4-6`, `claude-opus-4-8`, or + # `claude-fable-5` if a higher tier is warranted; SkillSpector + # passes it directly to the Anthropic API. + model: "claude-sonnet-4-6" # Per-skill verdict policy. v1 has one input (SkillSpector risk_score). # When more scanners join the pipeline we add new threshold fields here diff --git a/docs/CALIBRATION.md b/docs/CALIBRATION.md index 4e2879c..0833d1e 100644 --- a/docs/CALIBRATION.md +++ b/docs/CALIBRATION.md @@ -135,7 +135,7 @@ user-visible notification. Those 2 findings are real and minor; the cleanest fix is a one-line `echo` before each write in the upstream skill repo rather than any change here. -**Model swap caveat**: production runs against `claude-sonnet-4-5` +**Model swap caveat**: production runs against `claude-sonnet-4-6` via the Anthropic API (see "Provider choice" below), not against `gpt-4.1-mini`. The 25 → 2 delta above measures SkillSpector's LLM semantic pass *as a capability*; absolute counts may shift one or two @@ -163,9 +163,9 @@ scope. The contract documented here only takes effect once that edit lands (or is pasted by a human with workflow write access). Provider is `anthropic` against `api.anthropic.com` directly, model -`claude-sonnet-4-5-20250929`. The Anthropic API key is on a separate -billing line from Coder usage because SkillSpector cannot be routed -through Coder's AI Gateway today: +`claude-sonnet-4-6`. The Anthropic API key is on a separate billing +line from Coder usage because SkillSpector cannot be routed through +Coder's AI Gateway today: - aibridge does proxy Claude under its `/anthropic` path, but only in Anthropic's native `/v1/messages` shape. @@ -228,10 +228,10 @@ Re-run this analysis when any of: protects us from drifting silently; a deliberate bump should walk through this doc. - The LLM model or provider changes (e.g., moving from - `claude-sonnet-4-5` to Opus or to a non-Anthropic provider). - Different models filter differently; spot-check the five in-tree - skills before merging the provider swap and refresh the table - above. + `claude-sonnet-4-6` to Opus, Fable, or to a non-Anthropic + provider). Different models filter differently; spot-check the + five in-tree skills before merging the provider swap and refresh + the table above. - We observe a real-world skill that lands in an obviously wrong bucket (false positive or false negative). Open a tracking issue, link it from this doc, and adjust with evidence in the next PR. From 3170357f1a0a9caeb08c1f98f5f63e801d2e0e05 Mon Sep 17 00:00:00 2001 From: DevCats Date: Tue, 23 Jun 2026 21:20:48 +0000 Subject: [PATCH 13/13] chore: strip explanatory cruft from config.yaml and PR-state callouts config.yaml: collapse the llm: block to provider + model + a one-line flags: comment. The aibridge backstory, model justification, provider options table, and "contract with scan.yaml" prose all live in docs/CALIBRATION.md and the PR description; reproducing them next to the two values that actually matter is noise. README.md and docs/CALIBRATION.md: drop the two "this PR is in a weird in-between state" callout blocks. The PR description already covers the workflow split; the docs do not need to carry it. --- README.md | 17 ------------- config.yaml | 58 ++------------------------------------------- docs/CALIBRATION.md | 7 ------ 3 files changed, 2 insertions(+), 80 deletions(-) diff --git a/README.md b/README.md index 8a7cefc..75e528e 100644 --- a/README.md +++ b/README.md @@ -27,13 +27,6 @@ The registry site reads the public report through a small proxy endpoint in `coder/registry-server` (separate PR) and shows a per-skill badge. The registry's deploys are not gated on the scan result. -> **Note on the LLM semantic pass.** Step 3's conditional-LLM behavior -> requires the matching edit in `.github/workflows/scan.yaml`. That edit -> is part of the same change that introduces this section in the README; -> see the PR description for the diff. The current `scan.yaml` on this -> branch still hardcodes `--no-llm`, so adding the secret alone has no -> effect until the workflow edit also lands. - ## Reading the latest report Stable URLs, no auth required: @@ -91,16 +84,6 @@ scheduled scan publishes a useful result: optional `SLACK_WEBHOOK_URL` secret enables the `notify-slack-on-failure` job; without it that job is a no-op. -> **Workflow file note**: enabling LLM mode also requires the matching -> edit in `.github/workflows/scan.yaml` (it must export -> `ANTHROPIC_API_KEY` into the SkillSpector step, set -> `SKILLSPECTOR_PROVIDER=anthropic` and `SKILLSPECTOR_MODEL`, and -> conditionally append `--no-llm` when the secret is missing). That -> edit is part of the same change that introduces this section and -> is documented in the PR description, but is committed separately -> because the Coder Agents GitHub App on this repo currently lacks -> the `workflows: write` scope. - ## Repo layout ```text diff --git a/config.yaml b/config.yaml index 29805e5..fac4259 100644 --- a/config.yaml +++ b/config.yaml @@ -39,65 +39,11 @@ scanners: # so a bumper bot lives outside the loop until the upstream # publishes to PyPI and the pin can move into pyproject.toml. pin: "skillspector @ git+https://github.com/NVIDIA/SkillSpector.git@2eb844780ab163f01468ecf142c40a2ec0fcaec0" - # Extra CLI flags passed to every SkillSpector invocation. Left - # empty so .github/workflows/scan.yaml can drive --no-llm - # dynamically based on whether the LLM credential secret is set - # (see contract under the llm: block below). + # Empty so .github/workflows/scan.yaml can append --no-llm + # dynamically based on whether the LLM credential secret is set. flags: [] - # SkillSpector ships a two-stage analyser: fast static rules - # followed by an optional LLM semantic pass. Measured on the five - # in-tree skills, LLM mode dropped catalogue-wide findings from - # 25 to 2 and moved coder/setup from malicious to clean. See - # docs/CALIBRATION.md for the per-skill numbers and methodology. - # - # Contract with .github/workflows/scan.yaml: the workflow reads - # the credential matching the provider below from a repository - # secret. When the secret is set, LLM mode runs. When the secret - # is missing, the workflow appends --no-llm so a fresh fork is - # never broken by an unset secret. - # - # NOTE: scan.yaml in this branch still hardcodes --no-llm. The - # matching workflow edit is in this PR's description but is - # committed separately because the Coder Agents GitHub App on - # this repo currently lacks the `workflows: write` scope. After - # both land, the config and the workflow drive LLM mode together. - # - # Provider options: - # provider env var(s) endpoint - # anthropic ANTHROPIC_API_KEY api.anthropic.com - # openai OPENAI_API_KEY + OPENAI_BASE_URL any OpenAI-compatible URL - # nv_build NVIDIA_INFERENCE_KEY build.nvidia.com (free) - # - # Changing provider also requires updating the env block in - # .github/workflows/scan.yaml so the matching secret is wired in, - # and adding the secret under Settings > Secrets and variables > - # Actions. llm: - # Hits api.anthropic.com directly with an Anthropic API key (not - # routed through Coder's AI Gateway). aibridge was the original - # plan, but SkillSpector pipes every provider through - # langchain_openai.ChatOpenAI which hits `/v1/chat/completions`, - # and aibridge only mounts that path under `/openai`, not - # `/anthropic`. The `anthropic` provider in SkillSpector also - # hardcodes `https://api.anthropic.com/v1/` and ignores - # ANTHROPIC_BASE_URL, so aibridge cannot be steered into the - # Claude path today. This means the Anthropic API key is on a - # separate billing line from Coder usage. provider: anthropic - # claude-sonnet-4-6 is SkillSpector's own pick for the - # finding-classification task: it is the bundled default for - # the anthropic provider's `meta_analyzer` slot, which the - # provider docstring describes as "cheaper for the high-volume - # filter pass." SkillSpector's `DEFAULT_MODEL` is - # `claude-opus-4-6`, used for the deeper analyzer slots - # (~5x the per-token cost); the per-finding intent - # classification this scanner relies on does not need - # Opus-class reasoning. Bare alias rather than a dated suffix - # (the dated suffixes for the 4-6 series are not accepted as - # of June 2026; the bare alias is the canonical ID). Bump - # this string to `claude-opus-4-6`, `claude-opus-4-8`, or - # `claude-fable-5` if a higher tier is warranted; SkillSpector - # passes it directly to the Anthropic API. model: "claude-sonnet-4-6" # Per-skill verdict policy. v1 has one input (SkillSpector risk_score). diff --git a/docs/CALIBRATION.md b/docs/CALIBRATION.md index 0833d1e..b089bdf 100644 --- a/docs/CALIBRATION.md +++ b/docs/CALIBRATION.md @@ -155,13 +155,6 @@ secret is configured. The fallback to `--no-llm` is automatic when the secret is missing, so an unset secret on a fresh fork degrades the scan rather than breaking it. -**Important caveat for this PR**: `.github/workflows/scan.yaml` in the -current branch still hardcodes `--no-llm`. The matching workflow edit -is in the PR description but is committed separately because the Coder -Agents GitHub App on this repo currently lacks the `workflows: write` -scope. The contract documented here only takes effect once that edit -lands (or is pasted by a human with workflow write access). - Provider is `anthropic` against `api.anthropic.com` directly, model `claude-sonnet-4-6`. The Anthropic API key is on a separate billing line from Coder usage because SkillSpector cannot be routed through