The Evaluate Knowledge feature lets you bulk-test the accuracy and effectiveness of your AI Agent configurations. Using the Generate Q&A capability within AI Agent Studio, the system creates question and answer pairs from knowledge articles, validates agent responses, and integrates with the RAG Evaluation Framework for continuous improvement.

Steps to Setup Knowledge Evaluation

• Create a new AI Agent or click the Manage icon on an existing one.

  

• In the Manage window, click View next to Evaluate or select Evaluate Knowledge from the left pane.

• On the Knowledge Evaluations window, click the Caret (˅) icon next to the Generate Q&A button and select one of the following:

  • Generate Q&A: Automatically create relevant question answer pairs using Sprinklr AI.

  • Upload Q&A: Import Q&A pairs from an Excel file.

  • Add Q&A: Manually enter Q&A pairs.

• Click Save.

  

  Note: When using FAQ+, you can add multiple Q&A pairs and leave the Answer field blank. If no answer is provided, the system auto-generates one using the Generate Expected Answer feature a banner within the interface will indicate this.

Generate Q&A Automatically

When you select Generate Q&A, the system initiates an automated background job. No further action is required. While processing:

• The Generate Q&A button becomes disabled, and a loading indicator appears to show that the job is in progress. This prevents duplicate submissions and maintains system stability.

• A pop‑up window displays each step being executed for full transparency:

  • Extract knowledge articles: Identifies the relevant content sources.

  • Generate Q&A pairs: Creates questions along with their expected answers.

  • Validate alignment: Confirms the generated pairs correspond only to the appropriate knowledge content.

  • Save dataset: Stores the final Q&A pairs in the RAG Evaluation Framework.

Once processing completes, the generated Q&A pairs appear in the dataset view.

From the dataset view, you can:

• Review the generated pairs for relevance and accuracy.

• Edit the questions or answers to refine clarity, context, or tone.

• Export the dataset for evaluation workflows or integration with other systems.

​

Run and Review Knowledge Evaluations

Use Knowledge Evaluations to assess how well your AI Agent responds to test questions and to identify gaps in retrieval accuracy and response quality.

Run a Knowledge Evaluation

1. On the Knowledge Evaluations page, click Run Evaluation in the top‑right corner.

2. Allow the evaluation to complete.

Once the evaluation runs, results are generated based on the configured prompts and resources.

Export Evaluation Results

After the evaluation completes:

1. Click the Export icon in the top‑right corner.

2. Download the evaluation report in Excel format.

The exported Excel file includes detailed GTS (Golden Test Set) data, such as:

• Question

• Expected Answer

• Predicted Answer

• Answer with Citations

• Sources

• Context

• Reworded Question

• Overall Score

• Retrieval Precision

• Groundedness

• Answer Relevance

This report helps you analyze performance trends and identify areas for improvement.

View Evaluation Instructions (Optional)

• The View Instructions icon appears only after at least one evaluation has been executed.

• If no evaluation has been run, this option is not available.

Evaluation Metrics Description

The following metrics are included in the evaluation results:

• Question: The test query used to evaluate how the AI Agent responds.

• Expected Answer: The reference response defined in the Golden Test Set.

• Predicted Answer: The response generated by the AI Agent or LLM during evaluation.

• Overall Score: A combined score representing the overall response quality.

• Retrieval Precision: Measures how accurately relevant knowledge sources are retrieved.

• Groundedness: Indicates whether the response is supported by retrieved sources, reducing hallucinations.

• Answer Relevance: Measures how effectively the response addresses the question.

• Consistency: Measures whether the AI Agent produces stable responses across repeated evaluations

Configure Evaluation Prompt (Optional)

Before running an evaluation, you can update evaluation settings:

1. Enter a clear Prompt.

2. Add or select the relevant Resources.

3. Specify any Additional Fields, if required.

4. Click Save.

Running and reviewing Knowledge Evaluations helps you validate AI responses, improve content coverage, and continuously optimize AI Agent accuracy.