Scenarios are the foundation of AI Agent evaluation. Each scenario represents a customer interaction used to assess how effectively an AI Agent responds, aligns with business objectives, follows guardrails, and achieves the desired outcome.

Scenarios help validate agent behavior across a variety of customer situations before deploying changes to production. Scenarios can be:

• Automatically generated using agent configurations and historical data
• Manually created for targeted use cases
  

How Scenario Generation Works

Scenario generation uses inputs from the AI Agent configuration to create realistic test cases, including:

• Agent tasks and objectives
• Knowledge sources
• Guardrails and instructions
• Historical customer interactions
• Profile and case attributes
• Evaluation parameters and quality metrics

These inputs are combined to generate scenarios that reflect real customer conversations and provide broad behavioral coverage.

Scenario Sets

Scenarios are grouped into Scenario Sets, which act as collections for evaluation and simulation.

Examples:

• Billing Support Evaluation
• Product Troubleshooting
• Customer Onboarding

Grouping scenarios enables targeted evaluation of specific workflows, use cases, or customer journeys.

Create a Scenario

Access Scenarios

​

1. Open the AI Agent and navigate to Evaluate.

2. Select Scenarios from the left pane.

3. Click + Scenario.

   

Scenario Configuration

Window 1: Scenario Details

Define basic scenario information:

• Scenario Set: Select the destination scenario set
• Scenario Name: Enter a descriptive name

• Description: Define the interaction, including:

  • Customer issue or request

  • Emotional context

  • Expected outcome

  

Window 2: Scenario Generation

The system generates the following:

• Customer Persona: Customer characteristics
• Scenario: Full interaction context
• Strategy: Expected agent approach

Use Back to update inputs and regenerate results.

Window 3: Success Criteria

Automatically generated based on scenario inputs:

• Defines how success is measured

• May include:

  • Accurate issue identification
  • Correct resolution
  • Policy adherence
  • Professional communication
  • Minimal escalation

Success criteria are read-only. Update previous inputs to regenerate.

Click Save Scenario to complete.

Manage Scenario Sets

Available actions:

• Edit Scenario Set

  • Rename
  • Review scenarios

• Delete Scenario Set

  • Removes the set and its configuration

    

Bulk Scenario Generation

Generate multiple scenarios automatically to evaluate broader agent behavior.

Inputs Used

• Agent tasks and objectives
• Guardrails
• Knowledge sources
• Historical cases
• Evaluation parameters
• Profile and case attributes

Generate Scenarios

1. Click Generate Scenarios on the Scenario Sets page.

   ​

2. Configure:

   • Scenario Set Name

   • Date Range

   • Case Selection Conditions (Field, operator, values)

3. Filters allow targeting of:

   • Customer segments

   • Issue types

   • Channels

   • Products

   • Regions

Generation Process

The system creates scenarios using:

• Historical conversations
• Agent configuration
• Knowledge content
• Governance rules
• Evaluation parameters
• Customer and case attributes

Generated scenarios are added to the selected Scenario Set for review and use in simulations.