A/B testing answers one question: does this change actually improve user behaviour or not? Instead of relying on opinions like “this design looks better,” it validates changes using real user interactions, measurable outcomes, and controlled experimentation. It also reduces the risk of a change by testing on a subset of users before rolling it out to everyone.

A/B Testing is currently a Limited Availability (LA) feature. To enable it, raise a support ticket at tickets@sprinklr.com with the Live Chat application IDs where you want to run experiments.

Users need the “MANAGE LIVE CHAT EXPERIMENTS” permission enabled. Only users with this permission can create and update A/B testing experiments.

What You Can Change

• Theme: Application theme and trigger theme

• Home screen: Title, description, brand logo, cards

• Conversation screen: Title, description, input placeholder, welcome messages

What You Cannot Change

• Conversation disclaimer configuration

• Trigger icon conditions

• Header and chat actions

• Chat composer actions: Attachments, speech‑to‑text, location share, character limit

• Advanced features: Video calls, co‑browsing

• Advanced settings: One open conversation per user, persistent menu, attachment types, sound notification, sneak peek, chat labels by locale

Traffic split is the percentage of eligible traffic assigned to the control and each variant. Users are assigned probabilistically, so the actual split can vary at low volumes but gets closer to the configured percentages as traffic grows.

• Use a 50/50 split for two‑variant tests where both versions are equally trusted. This is the most statistically efficient setup.

• Use a 70/30 split when the new variant is riskier or unproven. This configuration requires more traffic to reach the same statistical power.

• Use a 33/33/34 split for three equally trusted variants, planning for a longer run since each variant accumulates data more slowly.

Once an experiment is running, audiences, weightages, and variants cannot be modified. Changing these settings during an active experiment can compromise the validity of the results. To make changes, stop the experiment, clone it, and launch a new experiment.

No. Once an experiment is ended there is no way to restart the same one. Instead, clone the experiment and start it again.

An experiment can only be in one status at a time:

• Draft: Created but not scheduled or started.

• Scheduled: Saved with a future start time and will auto‑start.

• Live: Actively running and assigning users to variants.

• Completed: Reached its end time or stopped manually and is read‑only.

If no end time is set, the experiment automatically ends after 180 days.

It depends on the field type.

• Non‑A/B‑testable fields
  Examples: turning off reply box attachments.

  • Changes automatically propagate to all live, scheduled, and draft experiments.

  • They do not apply to completed experiments.

  • Reason: these fields are always identical across control and variants, so updating them does not affect test integrity.

• A/B‑testable fields

  • Changes are not applied automatically to any experiment.

  • Draft and scheduled experiments whose control still carries the old value are flagged as “out of sync.”

  • Running and completed experiments remain unchanged.

An experiment is considered out of sync when its control no longer matches the current default Chat Application on at least one A/B‑testable field. This is a transparent flag, not an error.

• Affected experiments display an “Out of sync with default” chip on the records page and next to the control in the variant builder.

• To resolve the issue, open the experiment and click Sync. This action overwrites the affected control fields with the current default values.

• Syncing updates the control only. If the field is not one you intend to test, you must also update it manually in every variant. Otherwise, the leftover difference can skew results.

Updating the control or variants while an experiment is running would alter the comparison baseline during the test and compromise the validity of the results. To preserve result integrity, changes to the control and variants are not allowed once an experiment has started.

No. From the user’s perspective, they see one chat experience. They are not told they are in a test, and the experience behaves exactly like a normal Live Chat interaction.

There is no explicit opt-out from the user’s side. A/B testing only modifies supported widget elements and never removes or breaks core chat functionality, so the user always has a complete working experience.

No. A user is always in at most one experiment at a time. If several experiments are live and a user matches more than one audience, the system assigns them to a single experiment at random, and they keep seeing that one as long as they remain eligible. Users never see a blended experience mixing pieces of multiple variants.

Yes, this is called stickiness. Once assigned, the same variant is reused on future page loads and user updates as long as the experiment is still running and the user still matches its audience. This keeps the experience consistent and the reporting data clean.

For authenticated users, the assignment is sticky to the user identity: the same variant is reused across sessions, devices, and visits while the experiment runs and they remain eligible.

For anonymous users, the assignment is sticky to the browser session: the same variant is reused across page loads within that session.

Because anonymous stickiness is tied to the browser session, the user is treated as new on the next page load or user update. The assignment is recalculated from scratch, so they may or may not land in the same experiment or variant as before.

When a user logs in, an authenticated identity is passed to Live Chat. This counts as a user update, and the system immediately re‑evaluates assignment using the authenticated identity, without waiting for the next page load. If the user is now eligible for an experiment, they begin seeing its variant right away.

The same process applies in reverse when an authenticated user logs out and becomes a guest. The system re‑evaluates assignment based on the new identity and updates the experience accordingly.

Every experiment must have exactly one primary success metric, which is used to decide the winner.

• Write the hypothesis as: “If we change X, then Y will improve.”

• Y is the primary metric.

• The metric must be measurable, attributable to the variant, and meaningful to the business.

• Always choose the closest behavioral metric to Y, not a distant one.

  • Example: For a trigger icon test, the closest metric is “chat trigger clicked”, not CSAT.

Avoid testing more than one major change within a single experiment. Doing so prevents you from attributing the outcome to a specific cause.

• Keep one hypothesis per experiment.

• If you have multiple ideas, run them sequentially or set them up as mutually exclusive experiments by audience.

The key factor is sample size, not time. An experiment should continue until each variant has collected enough data to detect the effect you care about with statistical confidence.

Rules of thumb:

• Run for at least one full week (7 days) to balance out day‑of‑week effects.

• Avoid running beyond 8–12 weeks unless there is a strong reason.

• Do not stop early just because a variant appears to be winning. Early peeking is the most common reason A/B tests reach the wrong conclusion.

The minimum sample size depends on four inputs:

• Baseline rate (p)

• Minimum detectable effect (MDE)

• Significance level (α = 0.05)

• Statistical power (0.80)

For the default setup of 95% confidence and 80% power, use the pocket formula:

n per variant ≈ 16 × p(1 − p) ÷ d²

where d is the absolute lift (expected variant rate minus baseline).

Example:

• Baseline = 5%

• Relative lift = 10%

• Absolute lift d=0.005

• Required sample size n≈30,400 users per variant

A ready‑made reference table is available in the Design Guidelines document.

A 50/50 split is the most efficient because both variants accumulate data at the same rate. Any unbalanced split slows the experiment, since the smallest variant becomes the bottleneck.

The traffic penalty grows quickly:

• 60/40 split → about 4% more traffic required

• 70/30 split → about 19% more traffic required

• 80/20 split → about 56% more traffic required

• 90/10 split → about 178% more traffic required

An unbalanced split remains a valid risk‑control choice for unproven variants, but it comes with the cost of extra traffic.

This is almost always a sample size problem. Either keep the experiment running, or accept that the data does not yet support the conclusion and design a higher-traffic follow-up.

Treat segment-level differences as hypotheses for the next experiment, not as winners. A variant that wins only for mobile + logged-in users should be tested again as an audience-specific experiment.