Skip to content

Area: User guide

How A/B tests work

An A/B test in Releva is site-wide: for as long as it runs, a share of your visitors, the control group, gets no Releva, and the rest get Releva as usual. The details page then compares the two groups. You find it under Analytics › A/B Tests. This page explains what the test changes and how to read the numbers. For the click-by-click steps, read Start an A/B test and Check and stop an A/B test.

A test has no name, no segment and no choice of features. It is always the whole shop against the whole shop, with two settings:

Setting What it is Values
Control Group Fraction The share of visitors who do not get Releva. Slider from 0 to 10000 in steps of 1000; 10000 is 100 %. Default 5000 (50 %). The card on the list shows it in percent.
Hypothesis The question the significance test answers. One sided (default): does Releva do better than the control? Two sided: does Releva do better or worse?

Only one test can run per shop. The start date is the moment you confirm, and a running test cannot be edited: stop it and start a new one.

Each profile carries a random bucket number between 0 and 9999, drawn once. While a test runs, profiles whose bucket is below the control group fraction are the control group; all others are the treatment group. A visitor stays in the same group for the whole test. Visitors who have no bucket yet, for example because they gave no consent, are treated as treatment for personalisation and are left out of the results.

The control group:

  • sees no product blocks, banners or stories on your site;
  • gets the default search results instead of personalised ones in Search As You Type;
  • is excluded from every segment that campaigns and workflows use, so it receives no campaign messages. Transactional and triggered messages are still sent, with the configured product feeds.

The confirmation dialog states the same in short: fewer users see product blocks and banners, and fewer users enter your campaigns and receive your newsletters.

When you stop or delete the test, the control group fraction goes back to 0 and every visitor gets Releva again.

The results window starts the day after the test started and ends the day before it ended (or before today, while it runs). The first and the last day are always left out, because they are partial days. That means:

  • a test started today shows only zeros until its third day;
  • a running test never includes today’s traffic;
  • a test that ran for two days or less never shows anything.

The date picker on the details page, which only Releva staff see, is limited to the test period and applies the same rule.

Indicator Measures Counted from
Page Views By Group Page views per profile pageView events
Product Views By Group Product views per profile productView events
Revenue By Group Revenue per profile Your shop’s purchase conversion actions (orders sent to Releva)
Time On Page By Group Average active time per page view, in seconds SDK page engagement events
Scroll Depth By Group Average share of the page viewed, in percent SDK page engagement events
Purchases By Group, Average Order Value By Group Purchase rate and mean order value Releva staff only

The number of profiles per group is not counted from the events. Releva counts the unique profiles that viewed a page in the window and splits that total by the configured fraction. With 20 % control and 1000 profiles, the control pool is 200 and the treatment pool is 800. All per-profile rates use these pools.

Time on page and scroll depth are averages over tracked page views, not totals. A page shorter than the screen reports 100 % scroll depth straight away. If your storefront’s SDK does not send page engagement events, both stay at No Data.

Each indicator has three values:

  • Change: how much higher (or lower) the treatment group’s rate is than the control group’s, in percent of the control value. N/A when there is nothing to show: the change is exactly 0, or the control value is zero so no change can be computed.
  • Statistical significance: the confidence, from 0 % to 100 %, that the difference is real and not chance, computed for the hypothesis you chose. A one-sided test reports low confidence when the control wins; a two-sided test reports the size of the difference in either direction.
  • Status: Not yet significant while significance is 90 % or below; above 90 %, Releva wins when the change is positive and Releva doesn’t win when it is not.

The Influence column shows the change with a one-line verdict to Releva staff. Everyone else sees Test needs more data to reach significance there, whatever the numbers; use the card’s Change instead.

Profiles By Day at the bottom shows the modelled profile count per group per day. It has no table.

Stop a test when the indicators you care about have reached significance, or when the cost of keeping the control group without Releva outweighs what you learn. Stopping is final: the results stay on the ended test’s details page, where Delete removes them for good. Deleting a running test also ends it.