Simulations in AI Agent Evaluation
Updated
Simulations enable you to test and evaluate AI Agent performance using predefined scenarios and scenario sets. During a simulation, selected scenarios are executed against the AI Agent and assessed using configured evaluation metrics. This helps validate agent behavior, identify gaps, and measure readiness before deployment.
Evaluation Workflow
Simulations are part of the AI Agent evaluation workflow:
Configure the AI Agent
Generate Scenarios
Create Scenario Sets
Run Simulations
Review Evaluation Results
Simulation List View
The Simulations page displays all created simulations with the following details:
- Simulation Name
- Scenario or Scenario Set
- Status
- Created Date
Access Simulations
Create or open an AI Agent.
Navigate to Evaluate from the left pane.
Select Simulations.
Click + Simulation.
Create a Simulation
Configure Details
Provide the following information:
Simulation Name*: Enter a unique name to identify the simulation.
Description: (Optional) Add details to describe the simulation purpose.
Scenario Set*: Select the scenario set to use for evaluation.
Scenarios: Choose all or specific scenarios from the selected set.
Click Select, then Next.
Review Evaluation Metrics
- Review metrics configured for the AI Agent.
- Remove any metrics not relevant to the simulation.
Example: Focus on resolution metrics for issue handling or communication metrics for quality evaluation.
Click Start Simulation.
Simulation Status
During execution, simulations display the following statuses:
- Running
- Completed
- Failed
These indicators provide visibility into progress and results.
Launch Simulations from Scenarios
You can also initiate simulations directly from:
- Scenario records
- Scenario Set records
The simulation form is pre-filled based on the selected asset.
Evaluation Metrics
Simulations use configured metrics to evaluate performance, including:
- Agent Success Rate
- Containment
- Conversation Opening
- Issue Understanding & Probing
- Courtesy & Professionalism
- Empathy
- Active Listening
- Communication Clarity
- Ownership
- Issue Resolution
- Conversation Closure