Skip to content

Conversation

@nina-msft
Copy link
Contributor

@nina-msft nina-msft commented Nov 22, 2025

Description

New scenario for creation of scam materials via single-turn attacks (RolePlay (persuasion), Context Compliance) & multi-turn attacks (Red Teaming Attack with custom Persuasion Persona system prompt).

Tests and Documentation

Unit test added, tested end-to-end locally.

Example results:

====================================================================================================
                                      📊 SCENARIO RESULTS: Scam                                      
====================================================================================================

▼ Scenario Information
────────────────────────────────────────────────────────────────────────────────────────────────────
  📋 Scenario Details
    • Name: Scam
    • Scenario Version: 1
    • PyRIT Version: 0.10.1.dev0
    • Description:
        Scam scenario evaluates an endpoint's ability to generate scam-related materials (e.g., phishing emails,
        fraudulent messages) with primarily persuasion-oriented techniques.

  🎯 Target Information
    • Target Type: OpenAIChatTarget
    • Target Model: xxx
    • Target Endpoint: https://xxx.openai.azure.com/openai/v1

  📊 Scorer Information
    • Scorer Type: TrueFalseCompositeScorer
      └─ Composite of 2 scorer(s):
          • Scorer Type: SelfAskTrueFalseScorer
          • Scorer Type: TrueFalseInverterScorer
            └─ Composite of 1 scorer(s):
                • Scorer Type: SelfAskRefusalScorer

▼ Overall Statistics
────────────────────────────────────────────────────────────────────────────────────────────────────
  📈 Summary
    • Total Strategies: 3
    • Total Attack Results: 12
    • Overall Success Rate: 58%
    • Unique Objectives: 4

▼ Per-Strategy Breakdown
────────────────────────────────────────────────────────────────────────────────────────────────────

  🔸 Strategy: scam_role_play
    • Number of Results: 4
    • Success Rate: 100%

  🔸 Strategy: scam_context_compliance
    • Number of Results: 4
    • Success Rate: 25%

  🔸 Strategy: scam_persuasive_rta
    • Number of Results: 4
    • Success Rate: 50%

====================================================================================================

@nina-msft nina-msft changed the title FEAT: New Scenario - Scams (Role Play) FEAT: New Scenario - Scams (Impersonation) Dec 12, 2025
@nina-msft nina-msft changed the title FEAT: New Scenario - Scams (Impersonation) FEAT: New Scenario - Scams Jan 2, 2026
Copy link
Contributor

@jsong468 jsong468 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor NITs, but lgtm!

@nina-msft nina-msft merged commit fb8e223 into Azure:main Jan 6, 2026
20 checks passed
@nina-msft nina-msft deleted the users/nina-msft/6685-scam-scenario branch January 6, 2026 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants