# How A/B tests work

> What a Releva A/B test holds back, how visitors land in the control or treatment group, which days and events the results count, and how to read status, change and significance.

An **A/B test** in Releva is site-wide: for as long as it runs, a share of your visitors, the control group, gets no Releva, and the rest get Releva as usual. The details page then compares the two groups. You find it under **Analytics › A/B Tests**. This page explains what the test changes and how to read the numbers. For the click-by-click steps, read [Start an A/B test](https://releva.ai/documentation/guides/ab-tests/start-an-ab-test/) and [Check and stop an A/B test](https://releva.ai/documentation/guides/ab-tests/check-and-stop-an-ab-test/).

## One test, two settings

A test has no name, no segment and no choice of features. It is always the whole shop against the whole shop, with two settings:

| Setting                    | What it is                                       | Values                                                                                                                                      |
| -------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| **Control Group Fraction** | The share of visitors who do **not** get Releva. | Slider from **0** to **10000** in steps of **1000**; **10000** is 100 %. Default **5000** (50 %). The card on the list shows it in percent. |
| **Hypothesis**             | The question the significance test answers.      | **One sided** (default): does Releva do better than the control? **Two sided**: does Releva do better or worse?                             |

Only one test can run per shop. The start date is the moment you confirm, and a running test cannot be edited: stop it and start a new one.

## What the control group misses

Each profile carries a random bucket number between 0 and 9999, drawn once. While a test runs, profiles whose bucket is below the control group fraction are the **control group**; all others are the **treatment group**. A visitor stays in the same group for the whole test. Visitors who have no bucket yet, for example because they gave no consent, are treated as treatment for personalisation and are left out of the results.

The control group:

* sees no product blocks, banners or stories on your site;
* gets the default search results instead of personalised ones in Search As You Type;
* is excluded from every segment that campaigns and workflows use, so it receives no campaign messages. Transactional and triggered messages are still sent, with the configured product feeds.

The confirmation dialog states the same in short: fewer users see product blocks and banners, and fewer users enter your campaigns and receive your newsletters.

When you stop or delete the test, the control group fraction goes back to 0 and every visitor gets Releva again.

## Which days count

The results window starts the day **after** the test started and ends the day **before** it ended (or before today, while it runs). The first and the last day are always left out, because they are partial days. That means:

* a test started today shows only zeros until its third day;
* a running test never includes today’s traffic;
* a test that ran for two days or less never shows anything.

The date picker on the details page, which only Releva staff see, is limited to the test period and applies the same rule.

## What the indicators count

| Indicator                                                | Measures                                      | Counted from                                                    |
| -------------------------------------------------------- | --------------------------------------------- | --------------------------------------------------------------- |
| **Page Views By Group**                                  | Page views per profile                        | `pageView` events                                               |
| **Product Views By Group**                               | Product views per profile                     | `productView` events                                            |
| **Revenue By Group**                                     | Revenue per profile                           | Your shop’s purchase conversion actions (orders sent to Releva) |
| **Time On Page By Group**                                | Average active time per page view, in seconds | SDK page engagement events                                      |
| **Scroll Depth By Group**                                | Average share of the page viewed, in percent  | SDK page engagement events                                      |
| **Purchases By Group**, **Average Order Value By Group** | Purchase rate and mean order value            | Releva staff only                                               |

The number of **profiles** per group is not counted from the events. Releva counts the unique profiles that viewed a page in the window and splits that total by the configured fraction. With 20 % control and 1000 profiles, the control pool is 200 and the treatment pool is 800. All per-profile rates use these pools.

Time on page and scroll depth are averages over tracked page views, not totals. A page shorter than the screen reports 100 % scroll depth straight away. If your storefront’s SDK does not send page engagement events, both stay at **No Data**.

## Reading the results

Each indicator has three values:

* **Change**: how much higher (or lower) the treatment group’s rate is than the control group’s, in percent of the control value. **N/A** when there is nothing to show: the change is exactly 0, or the control value is zero so no change can be computed.
* **Statistical significance**: the confidence, from 0 % to 100 %, that the difference is real and not chance, computed for the hypothesis you chose. A one-sided test reports low confidence when the control wins; a two-sided test reports the size of the difference in either direction.
* **Status**: **Not yet significant** while significance is 90 % or below; above 90 %, **Releva wins** when the change is positive and **Releva doesn’t win** when it is not.

The **Influence** column shows the change with a one-line verdict to Releva staff. Everyone else sees **Test needs more data to reach significance** there, whatever the numbers; use the card’s **Change** instead.

**Profiles By Day** at the bottom shows the modelled profile count per group per day. It has no table.

## When to stop

Stop a test when the indicators you care about have reached significance, or when the cost of keeping the control group without Releva outweighs what you learn. Stopping is final: the results stay on the ended test’s details page, where **Delete** removes them for good. Deleting a running test also ends it.

## Next steps

* [Start an A/B test](https://releva.ai/documentation/guides/ab-tests/start-an-ab-test/): Choose the control group and the hypothesis to measure what Releva adds.
* [Check and stop an A/B test](https://releva.ai/documentation/guides/ab-tests/check-and-stop-an-ab-test/): Read the influence and significance of each indicator, then stop the test.
* [How product blocks work](https://releva.ai/documentation/learn/product-blocks/how-product-blocks-work/): Learn what the control group stops seeing on your shop.
* [How events get into Releva](https://releva.ai/documentation/learn/events/how-events-get-into-releva/): See where the events behind the test results come from.