Retries + DLQ in Events Architecture: Building a Resilient Online Store

In a modern digital marketplace, a single point of failure can lead to lost orders and dissatisfied customers. Event-driven architecture (EDA) provides the flexibility needed for high-scale systems, but it also requires a sophisticated error-handling strategy.

By utilizing a combination of automatic retries with backoff and Dead Letter Queues (DLQ), developers can ensure that temporary glitches don't become permanent failures.

1. The Happy Path: From Purchase to Fulfillment

The process typically begins with a user action, such as "Maria buys a toy in the online store". This triggers an initial event: OrderPlaced. The system then publishes a subsequent event, InventoryReservationRequested, which the Inventory Service listens for to discount stock from the database.

2. When Things Go Wrong: Handling Failures

Errors such as a database being down, network timeouts, or general service interruptions can cause the initial attempt to fail (InventoryReservationFailed). Instead of discarding the message, the system enters a retry phase.

Automatic Retry with Backoff + Jitter

To prevent overwhelming a recovering service, a "backoff + jitter" strategy is applied:

• 1st retry: Wait 1 second.

• 2nd retry: Wait 2 seconds plus a small amount of random "jitter".

• 3rd retry: Wait 4 seconds plus jitter.

Each of these attempts is tracked by a InventoryReservationRetrying event, providing full traceability and monitoring capabilities for the engineering team.

3. The Safety Net: The Dead Letter Queue (DLQ)

If all configured retries fail, the message is not lost. It is automatically moved to a Dead Letter Queue (DLQ), and an InventoryReservationDeadLettered event is published. The DLQ serves as a storage area that triggers alerts, notifying the technical team that manual intervention or a deeper investigation is required.

4. Recovery and Reprocessing

Once the underlying technical issue is resolved, the message in the DLQ can be reprocessed, either manually or through an automated script. This generates an InventoryReservationReprocessed event, allowing the order to re-enter the flow and successfully complete the inventory reservation.

Gift

The following image shows how it works:

Conclusion

A resilient online store must be prepared for the unexpected. By implementing a robust lifecycle of retries and utilizing a Dead Letter Queue as a final safety net, businesses can guarantee that every customer order is eventually processed. This architecture not only protects data integrity but also provides the monitoring tools necessary to maintain a high-performance system in the face of inevitable technical challenges.