This guide provides a practical, end‑to‑end framework for running Live Chat experiments. It explains how to frame a clear hypothesis, select the right success metric, size and schedule experiments based on sample size, and interpret results using Sprinklr reporting. The goal is to help brands separate real effects from noise, apply statistical checks with confidence, and make evidence-based decisions about whether to roll out, reject, or refine a variant.

This guide is organized into two main sections, each focusing on a critical stage of experimentation:

• Design Stage Guidelines: Focuses on setting up experiments with the right success metrics, hypotheses, guardrails, and duration planning.

• Experiment Analysis: Focuses on interpreting results, applying statistical checks, and following a structured decision flow to identify winning variants and set them as defaults.

​

Before starting, review the Glossary to understand the key terms used throughout.

​

Design Stage Guidelines

The design stage lays the foundation for a valid experiment. This section explains how to choose the right success metric, define hypotheses, and use guardrail metrics to protect the user experience. It also covers sizing experiments and determining duration based on traffic, audience, and traffic split. Following these guidelines ensures experiments are measurable, attributable, and meaningful to the business.

Key points:

• Define one primary metric to determine the winner.

• Pair with guardrail metrics (CSAT, AHT, order completion) to protect the wider experience.

• Test one major hypothesis per experiment.

• Size experiments by sample size, not elapsed time.

• Duration is driven by traffic volume, audience size, and traffic split.

• Avoid early stopping; run at least one full week, but not beyond 8–12 weeks unless necessary.

Choosing the Right Success Metric

Each experiment must define one primary success metric. This metric alone determines the winner, while supporting and guardrail metrics provide context but do not decide the outcome.

How to pick the primary metric

1. Write the hypothesis as 'If we change X, then Y will improve.'

2. Y is the primary metric. It must be measurable, attributable to the variant, and meaningful to the business.

3. Pick the closest behavioural metric to Y, not the furthest one. For a trigger icon test, the closest metric is 'chat trigger clicked', not 'CSAT'.

4. Confirm there is enough volume in this metric to reach minimum sample size in a reasonable time.

Examples of Hypothesis–Metric Pairing

Guardrail metrics

Always pair the primary metric with one or two guardrail metrics that protect you from 'winning' by harming the wider experience. Common guardrails: CSAT, AHT, or a business outcome like order completion.

• If the primary metric improved but CSAT dropped, the change is not safe to roll out.

• If the primary metric improved but AHT got significantly worse, the operational cost may outweigh the gain.

One hypothesis per experiment

Avoid testing more than one major change in a single experiment. If the result moves, you will not be able to attribute the change to a specific cause. If you have multiple ideas, run them sequentially or in parallel as mutually exclusive experiments by audience.

Experiment Duration

Experiments should be measured by sample size rather than duration. An experiment should run until it has collected enough data per variant to detect the effect you care about with statistical confidence. The recommended approach is to first estimate the minimum sample size you need per variant, and then convert that into a duration based on your application's traffic.

Key Drivers of Duration

• Traffic volume: How many eligible users land on the application per day. This is the single biggest driver of how long an experiment takes.

• Audience size: Narrower audiences mean fewer eligible users per day, so the experiment takes longer to reach minimum sample.

• Traffic split: The smallest variant determines duration. Once any variant has enough users, the others have already collected at least as much (subject to the unbalanced split penalty in section 3).

How to convert sample size into duration

1. Estimate baseline conversion rate of the metric you care about (for example, current chat trigger CTR is 5%).

2. Decide the minimum detectable effect (MDE), the smallest lift you would consider meaningful (for example, +10% relative lift).

3. Look up the minimum sample size per variant in the reference table or use the calculator described in Section 2.

4. Estimate daily eligible traffic for the experiment from your existing Audience Activity reports.

5. Divide minimum sample size by daily eligible traffic per variant to get the number of days needed.

6. Add a buffer for weekday/weekend effects.

Practical Rules for Running Experiments

• Run experiments for at least one full week (7 days). This evens out day-of-week effects.

• Avoid running experiments beyond 8–12 weeks unless there is a strong reason. After this point, results can lose validity because user behaviour, seasonality, traffic mix, and product context may change too much to trust the comparison.

• Never stop an experiment early just because it 'looks like' one variant is winning. Early peeking is the single biggest reason A/B tests reach the wrong conclusion.

Calculate Minimum Sample Size per Variant

Minimum sample size answers: How many users per variant do I need before I can trust the result? It depends on four inputs: the baseline rate of the metric, the smallest lift you want to detect, the significance level, and the statistical power.

​

The formula (Two-proportion Z-test)

For comparing two proportions (for example, CTR of trigger icon A vs. B, conversion rate, % of chats initiated), the standard test is the two-proportion Z-test.

n per variant ≈ ( z(α/2) + z(β) )² · ( p(1−p) + p'(1−p') ) / d²

• p = baseline conversion rate.

• p' = baseline × (1 + relative lift) = expected variant rate.

• d = p' − p = absolute lift.

• z(α/2) = 1.96 for α = 0.05

• z(β) = 0.84 for power = 0.80

Ready-to-use pocket formula

For the default setup of 95% confidence and 80% power, use this quick approximation:

n per variant ≈ 16 × p(1 − p) ÷ d²

• p = current baseline conversion rate.

• d = absolute lift = expected variant rate − baseline rate.

If the lift is given as relative lift, first calculate expected variant rate as p × (1 + relative lift), then calculate d.

Always use decimals in the formula. For example, 5% should be entered as 0.05.

Worked example: 5% baseline and 10% relative lift

Suppose the current baseline conversion rate is 5% and the smallest lift you want to detect is a 10% relative lift.

1. Convert the baseline into decimal form: p = 5% = 0.05.

2. Calculate the expected variant rate: p' = 0.05 × (1 + 10%) = 0.055.

3. Calculate the absolute lift: d = p' − p = 0.055 − 0.05 = 0.005.

4. Use the two-sided 95% confidence and 80% power shortcut: n per variant ≈ 16 × p(1 − p) ÷ d².

5. Substitute the values: n ≈ 16 × (0.05 × 0.95) ÷ 0.005².

6. This gives n ≈ 16 × 0.0475 ÷ 0.000025 = 30,400 users per variant. This is why the table shows 30,400 for the 5% baseline and 10% relative lift row.

Ready-made sample size reference

If you do not want to compute by hand, use the table below. It assumes α = 0.05 and power = 0.80 (the recommended defaults).

Heuristic: Low baselines plus small lifts require huge samples. If your baseline metric is rare (for example, 2% CTR) and you want to detect a small lift (for example, 5%), expect to run for weeks or even months.

Continuous metrics (T-test)

For continuous metrics like NPS, CSAT, or average session duration, use a two-sample T-test instead of the Z-test. The intuition is the same: you decide what difference you want to detect, the significance level and the power, and you compute the per-variant sample size needed.

How Traffic Split Affects Experiment Duration

The traffic split (weightage) directly affects how long an experiment will take. A 50/50 split is statistically the most efficient because both variants accumulate data at the same rate. Any unbalanced split slows the experiment down because the smallest variant becomes the bottleneck.

The unbalanced split penalty

The penalty factor for an unbalanced split is approximately:

Penalty ≈ 1 / (4 · w · (1 − w))

where w is the weight of one variant (between 0 and 1).

Common splits and their cost

Implications for multi-variant tests

• 33 / 33 / 34 is similarly efficient. Each variant just collects data more slowly because there are three of them.

• The duration is driven by whichever variant has the smallest weight, not the average.

• For risky or unproven variants, an unbalanced split (for example, 70 / 30) is a valid risk-control choice, accept the extra traffic cost in exchange for limiting exposure.

Experiment Analysis

After an experiment has run, the analysis stage determines whether a variant should be promoted, rejected, or refined. This section explains how to set up reporting, apply statistical checks, and follow decision rules. The goal is to provide a structured decision flow that helps teams confidently identify the winning variant and set it as the new default.

Key points:

• Use Sprinklr reporting (Audience Activity, Social Analytics) with filters for Experiment ID and Variant.

• Calculate conversion rates, lift, confidence intervals, and p‑values.

• Apply the two‑proportion Z‑test for binary outcomes; use a T‑test for continuous metrics.

• Declare a winner only if:

  • Minimum sample size is reached.

  • Primary metric difference is statistically significant (p < 0.05, CI excludes 0).

  • Guardrail metrics did not regress.

• Treat inconclusive or regressed outcomes as “no winner” and refine hypotheses before retesting.

• Once a winner is confirmed, set it as the default configuration.

Reporting for A/B Testing

Reporting for A/B testing is delivered through the following data sources in Care Reporting:

• Social Analytics

• Audience Activity

Two new dimensions are exposed everywhere variant-aware data flows: Experiment ID and Variant name. A new case-level field tags every case with the variant that produced it.

Supported Dimensions

• Experiment ID: To filter all reports down to a specific experiment.

• Variant ID: To break down any metric by variant.

• All the standard user-attribute dimensions already supported (URL, country, device, auth state, custom profile fields) for a segment wise deeper analysis

Sample reports

Below are the two canonical reports for an early-stage trigger icon experiment.

Data source: Social Analytics, filtered by experiment ID

Data source: Audience activity, filtered by experiment ID.

How to utilise Sprinklr reporting

1. Open the existing Audience Activity or Social Analytics dashboard.

2. Apply a filter on Experiment ID = your experiment.

3. Add Variant as a breakdown dimension to compare control vs test.

4. Add the primary metric that you want to compare variant performance on.

5. Add guardrail metrics (CSAT, AHT, custom business fields) as additional columns so you can read primary and guardrails on the same view.

6. When checking case level metrics, filter out cases tagged as polluted before drawing conclusions.

Generate standard experimentation report per variant

How to calculate confidence interval and p-value for each variant

For binary outcome metrics such as trigger clicked, conversation created, KB article clicked, or order completed, use a two-proportion Z-test to compare each non-control variant against the control. Calculate the statistics separately for every variant pair: Control vs Variant B, Control vs Variant C, and so on.

Inputs required

• n1 = sample size for control.

• x1 = conversions for control.

• n2 = sample size for the variant being compared.

• x2 = conversions for the variant being compared.

• p1 = x1 ÷ n1 = control conversion rate.

• p2 = x2 ÷ n2 = variant conversion rate.

Step 1: Calculate conversion rate and lift

Conversion rate for each variant is calculated as conversions divided by sample size.

Lift % is calculated as:

((p2 − p1) ÷ p1) × 100

A positive lift means the variant performed better than control on the primary metric; a negative lift means it performed worse.

Step 2: Calculate the confidence interval using the chosen confidence level

First choose the confidence level you want to report.

If the significance level is α, then confidence level = 1 − α.

For example, α = 0.05 corresponds to 95% confidence, while α = 0.01 corresponds to 99% confidence.

General CI formula for the absolute difference between two conversion rates:

CI absolute = (p2 − p1) ± z(α/2) × SE

where SE = sqrt((p1 × (1 − p1) ÷ n1) + (p2 × (1 − p2) ÷ n2)).

To express the interval as lift %, divide both the lower and upper bounds by p1 and multiply by 100.

For 95% confidence, use α = 0.05 and z(α/2) = 1.96. For other confidence levels, replace 1.96 with the corresponding z critical value (z(α/2)).

Step 3: Calculate the p-value using a two-proportion Z-test

General p-value formula for the two-proportion Z-test:

First calculate the pooled conversion rate p pooled = (x1 + x2) ÷ (n1 + n2).

Then calculate the pooled standard error:

SE pooled = sqrt(p pooled × (1 − p pooled) × ((1 ÷ n1) + (1 ÷ n2)))

Next calculate the z-score: z = (p2 − p1) ÷ SE pooled.

For a two-sided test, p-value = 2 × P(Z ≥ |z|),

where Z follows the standard normal distribution.

Decision rule: Compare the p-value with the chosen α. If p-value < α, the difference is statistically significant at the selected confidence level. If p-value ≥ α, do not declare a winner based on that metric.

Worked example using 95% confidence

Control A has 500 conversions out of 10,000 users, so p1 = 5.0%. Variant B has 600 conversions out of 10,000 users, so p2 = 6.0%. The lift is ((6.0% − 5.0%) ÷ 5.0%) × 100 = +20%.

Using the two-proportion Z-test, the z-score is approximately 3.11 and the p-value is approximately 0.0019. Since p-value < 0.05, the result is statistically significant. The 95% CI for absolute difference is approximately +0.37 percentage points to +1.63 percentage points, which translates to roughly +7.4% to +32.6% relative lift versus control.

Under this setup, a variant is statistically significant only when p-value < 0.05 and the confidence interval for the absolute difference does not include 0.

How to Decide the Winning Variant

Declaring a winner is more than just looking at which bar is taller. Use the following decision flow.

Winner decision flow

1. Confirm you have reached the minimum sample size on every variant. If not, keep the experiment running or accept that the result is inconclusive.

2. Confirm the primary metric difference is statistically significant — p < 0.05 (signifance level) and 95% CI excludes 0.

3. Confirm the guardrail metrics (CSAT, AHT, business outcomes) did not regress.

4. If all three are true, the variant with the higher primary metric is the winner.

5. If only the primary metric moved but a guardrail regressed, treat the result as 'do not roll out as-is' and consider a follow-up test that protects the guardrail.

6. If the primary metric did not move significantly, treat it as 'no winner'.

What 'no winner' really means

It does not mean the change is bad. It means there is no evidence it is better.

• Common causes: Change was too subtle, audience was too narrow, run was too short, hypothesis did not match the user need.

• Action: refine the hypothesis and design a stronger follow-up test rather than rolling out a change without evidence.

Setting the Winning Variant as Default

Once you have decided a winner, you can promote that variant's configuration to become the new default chat application. From that point onward, every visitor sees the winning experience and the experiment is closed out.

How to set it as default

1. Review the results of an ended experiment and identify the winning variant (see the A/B Testing - Reading Results guide).

2. Against the completed experiment on record page, click Set as Default on the experiment and choose the variant that is to be set as default app config.

3. Confirm. The entire configuration from that variant is copied onto the default chat application.

Glossary