Configure Evaluation in AI Sentiment Deployments
Updated
Evaluation helps you validate the output quality of an AI Sentiment predictions before or after deployment. You can evaluate new data or reuse completed bulk predictions, then inspect accuracy and request-level reasoning.
Prerequisites
Ensure that you have the following permissions enabled to initiate Evaluations:
- Run Auto Evaluations: You will be able to initiate Auto Evaluations for AI Sentiment deployment.
- Run Manual Evaluations: You will be able to initiate Manual Evaluations for AI Sentiment deployment.
Open Evaluation
- In the Deployments record manager, locate the AI Sentiment deployment.
- Select the vertical ellipsis (⋮) next to the deployment.
- Select Evaluation.
The Evaluation pane opens. The pane contains search, filter, sort, Auto Evaluations and Manual Evaluations tabs, and the add button. A tab appears only when the corresponding evaluation type is supported.
Run an Evaluation
- In the Evaluation pane, select Add (+) icon.
- Select Auto Evaluation or Manual Evaluation.
- Complete the Initiate Evaluation form.
- Select Save.
Type | Use |
Auto Evaluation | Automatically generates expected outputs and accuracy results. You can review the generated result and correct it when needed. |
Manual Evaluation | Lets a reviewer evaluate each request manually. |
Availability: If only one evaluation type is supported, selecting Add opens that form directly. If you have view access but not run permission, you can view existing evaluations but cannot initiate a new one.
Configure the Evaluation
Field | Description |
Name | Enter a clear, unique evaluation name. This field is required. |
Select Input Source | Select New Data or Existing Bulk Predictions. |
Evaluation Threshold | Enter the maximum number of records to evaluate. The supported maximum is 1,000. |
Search Bulk Predictions | For Existing Bulk Predictions, select one or more completed bulk prediction runs. |
New Data input | For New Data, select or upload input data by using the same input method available for bulk testing. |
Important: Only completed bulk predictions appear in the selection list. Select Save only after completing all required fields.
Monitor Evaluation Status
After you select Save, the evaluation appears in the Evaluation pane. Use search to find an evaluation by name, filter by status, or sort by triggered time or accuracy. Filters reset when you switch evaluation tabs.
Evaluation | Input | Status sequence |
Auto | New Data | Generating Predictions > Evaluation In-progress > Completed or Failed |
Auto | Existing Bulk Predictions | Evaluation In-progress > Completed or Failed |
Manual | New Data | Generating Predictions > Review In-progress > Completed or Failed |
Manual | Existing Bulk Predictions | Review In-progress > Completed |
When initiation succeeds, a confirmation message indicates that the evaluation started. Completion and failure notifications identify the deployment and provide the next action. For manual evaluation with new data, a notification appears when predictions are ready for review.
Understand Limits and Available Actions
A banner in the Auto Evaluations tab shows the quarterly allowance, remaining evaluations, and reset date. When the allowance is exhausted, the automatic evaluation option is unavailable until the date shown in the banner.
Point to an evaluation card to access these actions:
- View Details: Opens the original evaluation configuration in read-only mode.
- View Results: Opens results after an automatic evaluation is Completed. For a manual evaluation, this action becomes available when the status is Review In-progress.
Review Automatic Evaluation Results
- In the Auto Evaluations tab, locate an evaluation with Completed status.
- Point to the evaluation card, and select View Results.
- Review the summary metrics and select a request.
- Select the Auto Evaluation tab to compare the predicted output with the ideal output and review AI reasoning and insights.
Result | Meaning |
Original Accuracy | The accuracy calculated by the automatic evaluation before reviewer corrections. |
Evaluation Accuracy | The final accuracy after reviewer confirmations or overrides. |
Predicted Output | The sentiment returned by the deployed pipeline. |
Ideal Output | The expected sentiment generated by the evaluation process. |
AI Reasoning & Insights | The explanation for the ideal sentiment classification. |
Review and Correct Results
If review controls are available, mark an automatic evaluation as correct or incorrect. When you mark a result as incorrect, provide the expected sentiment and optionally add a comment. The system recalculates Evaluation Accuracy after the override. Use Undo to remove your own feedback and restore the unreviewed state.
Use request filters to narrow the list. Available filters can include existing request fields, Evaluation Overridden, and Accuracy. The request count updates to match the active filters.
Troubleshooting
Issue | Resolution |
Evaluation option is not visible | Confirm that automatic or manual evaluation is enabled for AI Sentiment and that you can view the deployment. |
Add is unavailable | Confirm that your role includes the required run permission. For automatic evaluation, also check the quarterly limit banner. |
Bulk prediction is missing | Confirm that the bulk prediction completed successfully. Incomplete runs are not listed. |
View Results is unavailable | Wait until an automatic evaluation reaches Completed or a manual evaluation reaches Review In-progress. |
Evaluation failed | Review the input and configuration, then initiate the evaluation again. |