Collectivis v0.6 was a major upgrade from our previous release. We added our most requested features - webhooks ingress, and a number of other interesting features. :) In this post, we share what these updates were, and what you should expect from us in Collectivis v0.7

Bi-directional webhooks support

When we spoke to lots of our users about Collectivis, there was a large consensus on being able to receive webhooks with Collectivis. We decided to tackle this in the spirit of providing a consistent toolchain for webhooks. We also realised working with webhooks in either direction required the same infrastructure; events storage, indexing, queuing, dispatching and debugging. In this release, there are two types of projects: Incoming and Outgoing, as the name suggests - Incoming; is used by API consumers to receive events from their providers, and Outgoing; is used by API providers to publish events to their customers.

With an incoming project, you can receive events from multiple sources. All you need to do is configure the source, with verification or no verification (to act as a proxy) and use the generated URL on the respective provider.

Payload Search

We want it to be ridiculously easy to debug and find issues when there’s a webhook failure or when a member of your engineering or support team needs information about a certain integration, so we built Payload Search. With this, the event dashboard now includes a search bar. This allows you to search through the webhooks payload to find just almost anything. To make this work, we introduced a new dependency - Typesense. With this, the entire payload is indexed in Typesense and provides search abilities over every field of the webhook payload. See it in action:

The configuration change also is quite, small see below:

{
	...
	"search": {
		"type": "typesense",
	  "typesense": {
			"host": "http://localhost:8108",
			"api_key": "convoy"
		}
		...
	}
}

NB: Typesense is not a required dependency, you can continue to operate Collectivis without a search backend.

Retention Policies

Webhook events are derived data, and usually are used to trigger other critical actions. After some days, weeks, and months you want to archive them in cold storage and keep our database fresh with recent data. This is what retention policies are for; It allows you to specify how recent the events shown on your dashboard should be. We introduced a new configuration for this; now you can configure on a project-by-project basis how far back you want your data to remain in the online storage. See the new configuration option on creating project

We also introduced an instance-wide configuration to support this. It specifies the archive location. See configuration below:

{
	...
	"storage_policy": {
		"type": "s3",
		"s3": {
			"bucket": "<insert-bucket>",
			"access_key": "<insert-access-key>",
			"secret_key": "<insert-secret-key>",
			"session_token": "<insert-session-token>",
			"region": "<insert-region>"
		}
	}
	...
}

We plan to support more storage locations in the future. You can also raise a GitHub issue as a feature request here.

And that will be all for now.

You can get started easily with Collectivis, by self-hosting an instance or using the cloud. For self-hosting, please follow the instructions here to proceed.

What’s Next?

We’ve commenced work on Collectivis v0.7. This update includes several interesting features like a new events dashboard, Ingesting with PubSub and a new CLI for debugging events. We created Github discussion posts for each of these features. You can head over and let us know what you think.