# How pipelines work

> What a pipeline forwards, the filter and transform functions, the payload your destination receives, how webhook, S3 and Google Cloud Storage outputs deliver and retry, and what the four counters mean.

A **pipeline** (**Data Platform › Pipelines**) is an outbound stream. It takes the events or profile changes that Releva records for your shop and forwards them, one by one, to a destination you own: a webhook, an Amazon S3 bucket or a Google Cloud Storage bucket. Use it to feed a data warehouse, a CRM or your own services with what Releva sees.

Pipelines send data **out** of Releva. To get customer data **in**, use the CSV import on **Data Platform › Profiles** ([Import profiles from a CSV file](https://releva.ai/documentation/guides/profiles/import-profiles-from-a-csv-file/)) or the [Profiles API](https://releva.ai/documentation/reference/profiles/overview/). For products, read [How your catalog gets into Releva](https://releva.ai/documentation/learn/catalog/how-your-catalog-gets-into-releva/).

## Pipelines, webhooks and exports

| Feature                                                                              | Sends                                                                                    | When                                         | Best for                                                     |
| ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | -------------------------------------------- | ------------------------------------------------------------ |
| **Pipeline**                                                                         | Every matching event or profile change, as it happens                                    | Continuously while switched on               | Mirroring Releva’s event stream or profile changes elsewhere |
| [**Webhook**](https://releva.ai/documentation/learn/webhooks/how-webhooks-work/) (Marketing Channels) | One request per profile, with the product recommendations of a campaign or workflow step | When a campaign or workflow reaches the step | Driving your own channel from a Releva audience              |
| [**Export**](https://releva.ai/documentation/learn/exports/how-exports-work/)                         | A file with results for a date range                                                     | Once, or on a schedule                       | Reports                                                      |

## What a pipeline does with each record

A running pipeline processes every record of your shop in this order:

1. **Type.** An **Event** pipeline receives events; a **Profile** pipeline receives every profile create and update. The type is fixed when the pipeline is created.
2. **Event list** (Event pipelines only). With **Events action** set to **Include**, only the events listed under **Events** pass. With **Exclude**, every event passes except the listed ones. The **Events** picker leaves out high-volume events (page views, page engagement, product views, searches, UTM session starts, campaign entries, and message sends and deliveries); an **Exclude** pipeline with an empty list forwards all of them. An **Include** pipeline with an empty list forwards nothing, although its card reads “Events including all”.
3. **Filter.** Your JavaScript function `(payload) => boolean` from the **Advanced filter** tab. A falsy result drops the record. The default `(payload) => { return payload ? true : false; }` keeps everything.
4. **Transform.** Your function `(payload) => anything` from the **Transform** tab. Its return value is what the destination receives. The default returns the payload unchanged.
5. **Output.** A `POST` to the webhook, or a line in the next file written to the bucket.

Filter and transform run in a sandbox with no network access and no access to Releva’s data beyond the payload. The transform can also use `URL` (reading `hostname`, `pathname`, `search` and the other properties), `btoa` and `atob`; the filter cannot. A function that throws counts the record as **Failed**. The code is not checked when you save, so test it on a copy of a real payload before switching the pipeline on.

Changes take effect within **one minute**: the worker reloads the list of running pipelines every 60 seconds. Saving the form never starts or stops a pipeline; only the switch on the list does. A new pipeline is always created switched off.

## The payload

The payload is the event or profile record as Releva stores it, with these fields normalised:

| Field                                         | Meaning                                                                                                                                                                                                     |
| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type`                                        | `event` or `profile`                                                                                                                                                                                        |
| `profileId`                                   | Your customer id, without Releva’s internal prefix                                                                                                                                                          |
| `action`                                      | Events only: the event key, such as `cartAdd` or `cartCheckoutSuccess`                                                                                                                                      |
| `timestamp`                                   | ISO 8601 in UTC, for example `2026-10-21T10:20:30.123Z`                                                                                                                                                     |
| `userAgent`                                   | Parsed into an object (`browser`, `os`, `source`, …); `{}` when absent                                                                                                                                      |
| `customString`, `customNumeric`, `customDate` | Arrays of `{ key, values[] }`; dates as ISO strings                                                                                                                                                         |
| `products`                                    | Events only: the products on the event, `[]` when none                                                                                                                                                      |
| `previous`                                    | Profile pipelines only: the previous values of the fields this write changed, or `null` for a brand-new profile. Compare it with the current values to react to a change, for example a new `subscribedAt`. |
| `domainId`                                    | Your shop id                                                                                                                                                                                                |

For the event keys and what each event records, see the [Events API overview](https://releva.ai/documentation/reference/events/overview/).

## Webhook delivery and retries

A webhook pipeline sends one `POST` per record to the **Output URL** with the transform result as a JSON body and these headers:

| Header                 | Value                                                                                                         |
| ---------------------- | ------------------------------------------------------------------------------------------------------------- |
| Your **Headers** rows  | As configured, one header per row                                                                             |
| `Content-Type`         | `application/json`                                                                                            |
| `X-Pipeline-ID`        | The pipeline’s id                                                                                             |
| `X-Message-ID`         | Internal message id of the batch the record arrived in                                                        |
| `X-Releva-Delivery-Id` | One UUID per record, the same on every retry. De-duplicate on it.                                             |
| `X-Releva-Attempt`     | `1`, `2`, …                                                                                                   |
| `Authorization`        | `Bearer <secret>` when an **Auth Secret** is selected. It replaces any `Authorization` row under **Headers**. |

Any `2xx` answer is a success. Delivery is **at least once**: a retry can repeat a request whose answer was lost, so make your endpoint idempotent on `X-Releva-Delivery-Id`.

**What is retried.** Timeouts, connection errors (`ECONNREFUSED`, `ECONNRESET`, `ETIMEDOUT`, temporary DNS failures) and the statuses `408`, `429`, `500`, `502`, `503` and `504`. Any other status (`400`, `401`, `403`, `404`, `422`, …) and a hostname that does not resolve are treated as permanent: the record is dropped and counted as **Failed**.

**How.** The first attempt waits up to 30 seconds. Two quick retries follow after 250 ms and 1 s with a 5-second timeout each. A record that still fails is parked and retried in the background every 30 seconds with a doubling delay (30 s, 1 min, 2 min, … up to 1 hour), with a 30-second timeout. After **8 attempts in total** the record is dead-lettered and dropped; dead records are kept for 30 days for support and then deleted. At most 2 000 parked records are kept per pipeline; beyond that, new failures are dropped. Parked records are retried only while the pipeline is switched on and still a webhook pipeline. Deleting the pipeline deletes them.

**Auth secrets.** Secrets live under **Settings › Shop & Integration › Secrets**. A static secret is sent as it is; a token-refresh secret is renewed before each call. A secret that is in use by a pipeline cannot be deleted; revoke it instead. Deliveries of a pipeline whose secret was revoked or cannot be refreshed fail and are dead-lettered after 8 attempts; Releva never sends them unauthenticated.

## S3 and Google Cloud Storage outputs

An **Amazon S3 Bucket** or **Google Cloud Storage Bucket** pipeline does not call you. It buffers records as JSON Lines (one JSON object per line), compresses them with gzip and writes a file to the bucket under your **Output URL** prefix (`s3://bucket/prefix` or `gs://bucket/prefix`):

```plaintext
<prefix>/<UTC midnight of the day, in epoch milliseconds>/<uuid>.jsonl.gz
```

A file is written every **30 seconds** when the buffer is not empty, or earlier when it reaches about 60 MB. Each object carries the metadata `message-count` and `pipeline-id` (S3) or `messageCount` and `pipelineId` (GCS) and the content type `application/gzip`. A failed upload drops that file; there is no retry for bucket outputs.

Bucket outputs use the shop’s own cloud credentials: the AWS account configured in **Settings** for S3 (**Output Region** sets the S3 region, for example `eu-west-1`) and the Google service account for GCS. **If the shop has no credentials for the chosen provider, nothing is written and no error is shown**; the card still counts the records as **Processed**. Configure the account before switching the pipeline on.

## The four counters

Each card on the **Pipelines** list shows four numbers for the selected period:

| Counter       | Counts                                                                                                                          | Note                                                                                                                                                |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Sent**      | Records of the shop offered to the pipeline, before the type, event list and filter                                             | A profile pipeline’s **Sent** also counts events, and the other way round, so it is a measure of the shop’s traffic, not of the pipeline’s matches. |
| **Captured**  | Records that passed the event list and the filter                                                                               | Shown with its share of **Sent**                                                                                                                    |
| **Processed** | Records transformed and handed to the output: posted to the webhook, or added to the bucket buffer                              | For bucket outputs this does not mean the file was uploaded.                                                                                        |
| **Failed**    | Captured records that were not processed: the filter or transform threw, or the webhook delivery failed or was parked for retry | Shown with its share of **Captured**                                                                                                                |

## Managing pipelines

* **Edit.** Open a card to change anything but **Type**. Saving returns you to the list and keeps the switch as it was.
* **Make a copy.** Opens a new pipeline prefilled from the current one, named “Copy of …”, switched off; nothing is saved until you select **Save**. With several shops you pick a **Target Domain** first. A copy into another shop keeps the **Auth Secret** reference, which that shop does not have, so clear it before saving.
* **Delete.** Asks **Are you sure?** and removes the pipeline and its parked deliveries.
* **Export** on the list creates a one-time export for the selected period under **Analytics › Exports** (see [How exports work](https://releva.ai/documentation/learn/exports/how-exports-work/)).
* Names are unique per shop. Pipelines are a paid add-on: locked buttons show **Unlock feature.**

There is no public API for pipelines; create and manage them in the admin.

## Next steps

* [Send events to a webhook with a pipeline](https://releva.ai/documentation/guides/pipelines/send-events-to-a-webhook-with-a-pipeline/): Forward selected events to your endpoint and switch the pipeline on.
* [How webhooks work](https://releva.ai/documentation/learn/webhooks/how-webhooks-work/): Compare pipelines with webhooks sent from a workflow.
* [How events get into Releva](https://releva.ai/documentation/learn/events/how-events-get-into-releva/): See where the events a pipeline forwards come from.
* [How customer profiles work](https://releva.ai/documentation/learn/profiles/how-profiles-work/): Learn what a profile record holds before you forward it.