Overview

The webhook processing system is designed to efficiently handle and process incoming webhooks using Redis streams. This modern architecture focuses on ensuring reliable message delivery and processing.

Workflow

1. Initialization:

   • The system starts by setting up a connection with a Redis server. Redis acts as a message broker, holding messages that need to be processed.

2. Receiving Messages:

   • The application reads messages from a Redis stream. These messages, representing webhooks, contain payload data that needs to be processed and sent to specific URLs.

3. Queueing Messages:

   • Messages received from the Redis stream are placed in a queue. This approach ensures that if there’s a sudden influx of messages, the system can manage them without getting overwhelmed.

4. Processing Messages:

   • The messages in the queue are then processed one by one. Each message is sent as an HTTP POST request to a specific webhook URL.

Components

• Redis Client:

  • Responsible for interacting with the Redis server, reading messages from the stream, and passing them onto the application.

• Queue:

  • Temporarily holds the messages before they are processed, acting as a buffer to manage the flow of messages.

• HTTP Client:

  • Takes each message from the queue and sends it as an HTTP POST request to a specified URL.

Concurrency Handling

• The system uses Go routines, allowing multiple operations to be performed concurrently. For example, while some messages are being added to the queue, others can be processed and sent out simultaneously.

• A buffered channel (queue) is used, which enables the application to handle multiple messages concurrently, ensuring that the system remains responsive and efficient.

Security

• The system can be configured to use SSL/TLS encryption when communicating with the Redis server, ensuring that messages are transmitted securely.

• The HTTP Client includes a header with a secret hash when sending messages. This feature allows the recipient of the message to verify its authenticity.

Data Structure

The data received from the Redis stream should follow this structure:

{
  "url": "http://example.com/webhook",
  "webhookId": "unique-webhook-id",
  "messageId": "unique-message-id",
  "data": {
    "key1": "value1",
    "key2": "value2"
  },
  "secretHash": "hash-value",
  "metaData": {
    "metaKey1": "metaValue1"
  }
}