Scorer composition (PR C, 3/3): complete LLM rollout with typed rubrics

[romanlutz]

Summary

Design: Scorer architecture refactor design gist

Final PR (3/3) of the scorer-architecture refactor ("Option 5 - scorer-owned composition"), following #2125 (ResponseHandler + internal round-trip helper) and #2146 (TrueFalse pure-composition API + CallableResponseHandler), both now merged to main.

This rolls the pure v1.0 composition API out to the remaining LLM-based scorers and removes the last transitional forwarder. Each migrated scorer owns its target, prompt configuration, and ResponseHandler, then calls _run_llm_scoring_async(...) directly instead of the deleted Scorer._score_value_with_llm_async base-class forwarder.

The final API also introduces typed rubric/configuration objects. Factory methods render the LLM prompt and configure runtime validation, categories, and normalization from the same object, preventing hidden defaults from drifting away from prompt content. Direct custom construction requires the final prompt together with its rubric/configuration.

Scorers migrated

• SelfAskQuestionAnswerScorer
• SelfAskRefusalScorer
• SelfAskCategoryScorer
• SelfAskGeneralTrueFalseScorer
• SelfAskScaleScorer
• SelfAskLikertScorer
• SelfAskGeneralFloatScaleScorer
• InsecureCodeScorer

Final composition API

• SelfAskScaleScorer.from_scale(...) uses a NumericRubric; prompt rendering, category, and normalization bounds come from that rubric. Omitting it retains the bundled tree-of-attacks rubric.
• SelfAskLikertScorer.from_likert_scale(...) uses an immutable LikertScale; configured entries define prompt text, valid returned levels, category, bounds, and normalization.
• SelfAskGeneralFloatScaleScorer requires a NumericRange for explicit normalization bounds and optional category.
• SelfAskCategoryScorer.from_content_classifier(...) uses a ContentClassifier; configured categories and boolean relationships are validated inside the retry boundary.
• SelfAskTrueFalseScorer.from_question(...) renders from a TrueFalseQuestion; custom direct construction requires both prompt and question, while omitting both retains the bundled default.
• InsecureCodeScorer.from_harm_categories(...) uses the same immutable category snapshot for prompt rendering and score metadata.
• Prompt templates can be bundled defaults, SeedPrompt instances, or inline Jinja strings where supported. Module-level render_*_system_prompt helpers expose rendering without constructing a scorer.

Other details

• Float-scale scorers set numeric_value=True on their default JsonSchemaResponseHandler. Embedded response schemas flow through the default handler; caller-supplied handlers continue to own their own response contract.
• Likert entries must be unique and strictly increasing, and model responses must exactly match a configured integer level.
• Typed rubric collections are immutable or defensively snapshotted to keep scorer identifiers and runtime behavior stable.
• First-party consumers and executable docs were migrated, including Crescendo, Tree of Attacks, the fuzzer, fairness benchmark examples, the psychosocial scenario, scorer initializers, and paired Jupytext notebooks.
• Documentation examples now show the final composition API and synchronized .py / .ipynb representations.

Breaking changes (v1.0)

Legacy path-based constructor arguments and transitional helpers are removed in favor of explicit composition constructors and from_* factories. Notably, from_scale_arguments / load_scale_arguments are replaced by from_scale(...) with NumericRubric; the minimal normalization type is NumericRange. No deprecation shims remain.

Testing

• Full unit suite: 10,272 passed, 120 skipped
• Changed-file Ruff formatting/checks, notebook linting, pre-commit hooks, and ty checks pass
• Six affected executable notebooks were re-run successfully with refreshed outputs
• The Azure SQL attacks notebook retains its previous output because the retired demo database hostname no longer resolves

[rlundeen2]

Can you link you design gist? (update: found it, but it is convenient to have here :))