This document explains how A/B testing experiments move through their lifecycle and how default chat application configuration changes are handled across draft, scheduled, live, and completed experiments. It also covers sync rules for A/B-testable and non-A/B-testable fields, out-of-sync states, and the recommended handling for running experiments.

Before starting, review the Glossary to understand the key terms used throughout.

Experiment Lifecycle at a Glance

Each experiment is always in one specific status. The status determines which actions you can take and how the experiment behaves when configuration changes happen.

​

Status Transition Rules

Sync Rule for Non-A/B-Testable Fields

Non-A/B-testable fields cannot be tweaked inside any variant by design. The control and every variant always carry the same value for these fields. So, when the default app config is changed with a non-A/B-testable field, it gets updated in all experiments variants but not for ones in completed status.

Behaviour

• Change in default propagates automatically to every live, scheduled and draft experiment.

• Completed experiments are read-only and are not updated.

Why this is safe

Because these fields are identical across the control and all variants of every experiment, propagating the change does not introduce any difference between variants. It cannot create artificial lift as well, so it will not disrupt reporting, and it does not change variant behaviour relative to each other.

​

Example: Turning off chat composer attachments

A brand updates the default chat application to disable file attachments in the chat composer. Since this is a non-A/B-testable field, every live, scheduled and draft experiment on that application picks up the change immediately on the next save. Customers on every variant and customers on the default experience stop seeing the attachment option from that point on. Experiments that are already ended are not updated with this configuration change.

Sync Rule for A/B-Testable Fields

A/B‑testable fields are the ones you intentionally vary across different variants. To protect experiment integrity, the system enforces strict rules: editing an A/B‑testable field in the default configuration will never silently overwrite a live test or pollute its results.

What happens when you update default app config on an A/B-testable field

1. You edit an A/B testable field and update the default chat application.

2. Nothing is changed in any experiment. Only your default app will be updated

3. Draft and scheduled experiments whose control still carries the old default value are now considered out of sync for that field.

4. These experiments are surfaced on the records page with an 'Out of sync with default' chip.

5. You can sync them individually from the experiment page, or accept the divergence if intended and proceed.

Why running experiments can never be updated

• Updating the control of a running experiment alters the comparison baseline while the test is in progress.

• Locking active experiments against such changes ensures the results remain valid and trustworthy.

Why completed experiments are not updated

A completed experiment is a historical record. After completion, they remain locked in their original state, and subsequent default configuration changes, whether A/B testable or non‑testable, are not propagated.

The Out-of-Sync State

An experiment is out of sync when its control no longer reflects the current default chat application on at least one A/B-testable field. This is not an error but it is a transparent way to show that the baseline has moved since the experiment was created or last synced.

Where it is shown

• On the Experiments records page, each affected experiment shows an 'Out of sync with default' chip in its row.

• Opening the experiment also shows the chip on the variant builder near the control.

How to resolve it

1. Open the experiment from the records page.

2. Click Sync button to overwrite the affected fields of control variant with the current default values.

3. Once confirmed, the 'out of sync' chip clears.

Resolution at start time

• If your scheduled experiment is out of sync or you try to manually start an out of sync experiment - the system does not block auto-start on sync state.

• If you do not want it to start out of sync, sync the control version before the scheduled time or before you start it manually.

• If you start it without syncing, the experiment will run on a stale application config, and that stale config might also carry forward to the default app once you promote the winning variant at the end of the experiment.

​Glossary

​

​

​

​

​