Custom Evaluation Metrics enable organisations to define business-specific criteria for evaluating AI Agent conversations. Unlike standard evaluation metrics, custom metrics allow administrators to create tailored evaluation questions that align with their unique quality and performance requirements.

Each custom metric uses an AI-powered evaluation model to assess conversations based on a defined evaluation question and supporting instructions. These configurations help ensure consistent and objective evaluation of AI Agent interactions.

Once created, a custom metric can be updated and refined without requiring the creation of a new metric.

Create a Custom Evaluation Metric

1. Open AI Agent Builder and navigate to Evaluate from the left navigation pane.

2. On the Evaluation Metrics page, click + Evaluation Metric in the upper-right corner.

3. In the Create Evaluation Metric window, enter the Metric Name and Description.

4. Under Model Details, provide the Model Question and Model Description.

Understanding the Configuration Fields

• Metric Name: A unique name used to identify the evaluation metric throughout the Evaluation module.
• Description: A brief explanation of the metric's purpose and what it measures.

• Model Question: Defines the specific question used to evaluate a conversation. The question should clearly describe the behaviour, outcome, or interaction being assessed.

  Example questions:

  • Did the AI Agent greet the customer professionally?
  • Did the AI Agent provide the correct resolution?
  • Did the AI Agent offer additional assistance before ending the conversation?
• Model Description: Provides detailed guidance on how the conversation should be evaluated. Well-defined descriptions help ensure consistent assessments by outlining expected behaviours, acceptable responses, and any additional evaluation criteria.

Click Save.

The new custom evaluation metric becomes available for use in AI Agent evaluations.

Example Configuration

Edit a Custom Evaluation Metric

You can update an existing custom metric whenever your evaluation requirements change.

1. Open AI Agent Builder and navigate to Evaluate.
2. On the Evaluation Metrics page, click the Options icon next to the required custom metric.
3. Select Edit Evaluation Metric to modify the metric name, description, or model details.
4. Select Metric Configuration to update the evaluation criteria and refine how the metric assesses conversations.
5. The Edit Metric Configuration page allows you to:
• Refine the evaluation criteria.
• Adjust the metric's assessment logic without creating a new metric
• Click Save to apply the changes.