> ## Documentation Index
> Fetch the complete documentation index at: https://getconvoy.io/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Retention Policy

Convoy supports exporting events to any S3 compatible cold storage.

> **Note:**
>
> * The retention policies functionality is available only for licensed users.
> * This feature is currently in beta and disabled by default. To enable it, use the environment variable `CONVOY_ENABLE_FEATURE_FLAG=retention-policy` or the CLI flag `--enable-feature-flag=retention-policy`.

## Configuring Retention

### JSON Configuration

You can configure read replicas by adding the `storage_policy` object to your database configuration JSON.

| Key                | Description                                               |
| ------------------ | --------------------------------------------------------- |
| `type`             | Storage type, either "s3" or "on\_prem"                   |
| `s3.prefix`        | Prefix for S3 storage path                                |
| `s3.bucket`        | Name of the S3 bucket                                     |
| `s3.access_key`    | AWS access key for S3 authentication                      |
| `s3.secret_key`    | AWS secret key for S3 authentication                      |
| `s3.region`        | AWS region where the S3 bucket is located                 |
| `s3.session_token` | Temporary session token for AWS authentication (optional) |
| `s3.endpoint`      | Custom endpoint URL for S3-compatible storage             |
| `on_prem.path`     | File system path for on-premises storage location         |

```json using storage policy via JSON configuration theme={null}
{
	"storage_policy": {
		"type": "s3 | on_prem",
		"s3": {
			"prefix": "<insert-s3-prefix>",
			"bucket": "<insert-s3-bucket>",
			"access_key": "<insert-s3-access-key>",
			"secret_key": "<insert-s3-secret-key>",
			"region": "<insert-s3-region>",
			"session_token": "<insert-s3-session-token>",
			"endpoint": "<insert-s3-endpoint>"
		},
		"on_prem": {
			"path": "<insert-on-prem-path>"
		}
	}
}
```

### Environment Variables Configuration

Alternatively, you can supply environment variables to configure the storage policy.

| Environment Variable               | Description                                               |
| ---------------------------------- | --------------------------------------------------------- |
| `CONVOY_STORAGE_POLICY_TYPE`       | Storage type, either "s3" or "on\_prem"                   |
| `CONVOY_STORAGE_AWS_PREFIX`        | Prefix for S3 storage path                                |
| `CONVOY_STORAGE_AWS_BUCKET`        | Name of the S3 bucket                                     |
| `CONVOY_STORAGE_AWS_ACCESS_KEY`    | AWS access key for S3 authentication                      |
| `CONVOY_STORAGE_AWS_SECRET_KEY`    | AWS secret key for S3 authentication                      |
| `CONVOY_STORAGE_AWS_REGION`        | AWS region where the S3 bucket is located                 |
| `CONVOY_STORAGE_AWS_SESSION_TOKEN` | Temporary session token for AWS authentication (optional) |
| `CONVOY_STORAGE_AWS_ENDPOINT`      | Custom endpoint URL for S3-compatible storage             |
| `CONVOY_STORAGE_PREM_PATH`         | File system path for on-premises storage location         |

```shell using storage policy via environment variable configuration theme={null}
CONVOY_STORAGE_POLICY_TYPE=''
CONVOY_STORAGE_AWS_PREFIX=''
CONVOY_STORAGE_AWS_BUCKET=''
CONVOY_STORAGE_AWS_ACCESS_KEY=''
CONVOY_STORAGE_AWS_SECRET_KEY=''
CONVOY_STORAGE_AWS_REGION=''
CONVOY_STORAGE_AWS_SESSION_TOKEN=''
CONVOY_STORAGE_AWS_ENDPOINT=''
CONVOY_STORAGE_PREM_PATH=''
```

## Retention Using Postgres Partitions

We introduced a new way to manage retention using Postgres-native partitions
using [go\_partman](https://github.com/jirevwe/go_partman) with each table is partitioned by day.
Prior to `Convoy v25.1.1` the job that ran the retention policy carried out two tasks:

* Running `SELECT` queries and uploading webhook event data to S3.
* Running `DELETE` queries on the tables.

### Problems with the old system

#### Concurrency

The problem with the old approach was that both the backup and deletion queries ran in the same job,
and any of these tasks could fail for a number of reasons:

* The backup action could fail because of a network issue, making the retention query not run.
* The backup action could succeed, but the retention query would time out leading to duplication data the next time the backup action ran.

This approach also had some other issues:

* There also wasn't any visibility into what error occurred or why the backup did not work.
* Running `DELETE` queries on moderately large Postgres databases triggers `AUTOVACUUM` and causes disk pressure.

#### Retention Granularity

The previous system also didn't make it possible to specify different retention periods for different projects.
Now it is possible to specify different retention periods per project,
and this can be managed from the project settings page.

### Proposed Solution

With the new system, the backup and delete operations run in different jobs that are triggered at different times.

#### Backups

The backup job backs up events sent on a particular day at the end of that day.

#### Deletion

The retention (deletion) job deletes events sent on a particular day n-days after it was sent;
where n is the retention policy duration.

## Switching to Partition for Retention

The tables that are partitioned are:

* `convoy.delivery_attempts`
* `convoy.event_deliveries`
* `convoy.events`
* `convoy.events_search`

Before partitioning (or un-partitioning), the tables be sure to back up the database,
or test it on a fresh database with all the organization and project configuration copied over.

### Partitioning the tables

To switch to managing retention using Postgres partitions, you need to run:

```shell partition all tables theme={null}
# The retention policy feature flag must be enabled
convoy utils partition --enable-feature-flag=retention-policy
CONVOY_ENABLE_FEATURE_FLAG=retention-policy convoy utils partition
```

> If you are running convoy in a docker container, you can execute this command using `docker exec`
>
> ```shell partitioning using docker exec theme={null}
> docker exec -it convoy-staging /cmd util partition
> ```

When the above command is run, it attempts to partition all the tables at the same time.
This operation can take a while (10 to 60 minutes) if the tables are moderately
large, so you would need to partition them one after the other like this:

```shell partition tables one after the other theme={null}
convoy utils partition delivery_attempts
convoy utils partition event_deliveries
convoy utils partition events
convoy utils partition events_search
```

### Un-partitioning the tables

To switch back from managing retention using Postgres partitions,
you can un-partition all of them at the same time by running:

```shell unpartition all tables theme={null}
convoy utils unpartition
```

Or you can un-partition them one after the other by running:

```shell unpartition tables one after the other theme={null}
convoy utils unpartition delivery_attempts
convoy utils unpartition event_deliveries
convoy utils unpartition events
convoy utils unpartition events_search
```