Outbox Pattern

How to guarantee reliable event publishing in distributed systems using a transactional outbox.

What problem it solves

In event-based systems, a microservice needs to do two things when it processes an operation: save data to its database and publish an event to notify other services. The problem is that these are two operations across two different systems (the database and the message broker), and there is no atomic transaction that covers both.

The dual write problem

Order Service:
  1. Save order in PostgreSQL  ✅
  2. Publish "OrderCreated" to Kafka  ❌ (network failure)

Result: The order exists in the database,
but no other service found out about it.

This scenario is called a dual write and it can cause:

• Data saved, event lost: The order exists but inventory was never reserved
• Event published, data not saved: Other services react to an order that does not exist
• Silent inconsistency: The system appears to work but data across services diverges

Trying to solve this with “publish first, save later” only reverses the problem. And using distributed transactions (2PC) introduces latency and points of failure.

Real-world scenarios where it happens

How it works

The Outbox Pattern solves the dual write by using a single database transaction for both operations. Instead of publishing the event directly to the broker, the service writes the event to an outbox table within the same transaction that saves the business data.

Basic flow

┌─────────────────────────────────────────────┐
│  Database transaction (atomic)              │
│                                             │
│  1. INSERT INTO orders (...)                │
│  2. INSERT INTO outbox (event, payload...)  │
│                                             │
│  COMMIT                                     │
└─────────────────────────────────────────────┘
         │
         │  (separate process)
         ▼
┌─────────────────────┐     ┌──────────────┐
│  Relay / Publisher   │────►│  Kafka /     │
│  (reads outbox table)│     │  RabbitMQ    │
└─────────────────────┘     └──────────────┘

The key is that both writes (business data + event) happen in the same database transaction. If the transaction fails, neither one is persisted. If it succeeds, both are persisted. There is no window of inconsistency.

Structure of the outbox table

CREATE TABLE outbox (
  id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  aggregate_type  VARCHAR(255) NOT NULL,  -- "Order", "Payment"
  aggregate_id    VARCHAR(255) NOT NULL,  -- aggregate ID
  event_type      VARCHAR(255) NOT NULL,  -- "OrderCreated"
  payload         JSONB NOT NULL,         -- event data
  created_at      TIMESTAMP NOT NULL DEFAULT NOW(),
  published_at    TIMESTAMP NULL          -- NULL = pending
);

-- Index for the publisher
CREATE INDEX idx_outbox_pending
  ON outbox (created_at)
  WHERE published_at IS NULL;

Usage example in code

// Inside a database transaction
BEGIN;

  -- 1. Business operation
  INSERT INTO orders (id, customer_id, total, status)
  VALUES ('ord-001', 'cus-789', 150.00, 'confirmed');

  INSERT INTO order_lines (order_id, product_id, quantity, price)
  VALUES ('ord-001', 'prod-42', 2, 75.00);

  -- 2. Event in the outbox table (same transaction)
  INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload)
  VALUES (
    'Order',
    'ord-001',
    'OrderConfirmed',
    '{"orderId": "ord-001", "customerId": "cus-789", "total": 150.00}'
  );

COMMIT;
-- Both writes are atomic: either both are saved, or neither is

Publishing strategies

There are two main ways to read the outbox table and publish the events to the broker:

Polling Publisher

A periodic process queries the outbox table looking for unpublished events, sends them to the broker, and marks them as published.

┌──────────────┐                    ┌──────────┐
│   Outbox     │  SELECT * FROM     │ Polling  │
│   Table      │◄─── outbox WHERE ──│ Publisher│
│              │  published_at NULL  │          │
└──────────────┘                    └────┬─────┘
                                        │
                                   Publish event
                                        │
                                        ▼
                                   ┌──────────┐
                                   │  Broker  │
                                   └──────────┘

Advantages: Simple to implement, requires no additional infrastructure. Disadvantages: Latency (depends on the polling interval), load on the database.

Typical polling implementation:

// Every 500ms (configurable)
loop {
  events = SELECT * FROM outbox
            WHERE published_at IS NULL
            ORDER BY created_at
            LIMIT 100
            FOR UPDATE SKIP LOCKED;  -- avoids conflicts between publishers

  for event in events {
    broker.publish(event.event_type, event.payload);

    UPDATE outbox
    SET published_at = NOW()
    WHERE id = event.id;
  }
}

Change Data Capture (CDC)

A CDC connector (such as Debezium) monitors the database transaction log and automatically captures inserts into the outbox table.

┌──────────────┐                    ┌──────────┐
│  PostgreSQL  │  WAL / binlog      │ Debezium │
│  Transaction │──────────────────► │  (CDC)   │
│  Log         │                    └────┬─────┘
└──────────────┘                         │
                                    Publish event
                                         │
                                         ▼
                                    ┌──────────┐
                                    │  Kafka   │
                                    └──────────┘

Advantages: Very low latency (near real-time), does not add extra load with queries. Disadvantages: Requires additional infrastructure (Debezium, Kafka Connect), more complex to operate.

Comparison of strategies

Delivery guarantees

The Outbox Pattern guarantees at-least-once delivery. This means an event may be published more than once. That is why consumers must be idempotent.

Duplication scenario:
  1. Publisher reads event from outbox
  2. Publisher publishes event to Kafka  ✅
  3. Publisher fails before marking it as published  ❌
  4. Publisher restarts and publishes the same event again

Solution: The consumer detects the duplicate by the event ID
and ignores it (Idempotency Pattern).

Cleaning up the outbox table

The outbox table grows continuously. You need a cleanup strategy:

-- Option 1: Delete events published more than 7 days ago
DELETE FROM outbox
WHERE published_at IS NOT NULL
  AND published_at < NOW() - INTERVAL '7 days';

-- Option 2: Partitioning by date (more efficient)
-- Create monthly partitions and drop old partitions
CREATE TABLE outbox (
  ...
) PARTITION BY RANGE (created_at);

-- Option 3: Move to an archive table before deleting
INSERT INTO outbox_archive SELECT * FROM outbox
WHERE published_at < NOW() - INTERVAL '30 days';

Advantages

• Guaranteed consistency: Data and events are written in the same transaction, eliminating the dual write
• No distributed transactions: You do not need 2PC or XA transactions
• Natural auditing: The outbox table works as a log of all published events
• Resilience: If the broker is down, events accumulate and are published when it comes back
• Temporal decoupling: The service does not depend on the broker’s availability
• Compatible with any relational database: You only need one additional table

Trade-offs / Disadvantages

• Additional latency: Events are not published instantly (especially with polling)
• Operational complexity: You need to maintain the publisher or the CDC connector
• The outbox table grows: You need a periodic cleanup strategy
• At-least-once: Consumers must handle duplicates (idempotency is mandatory)
• Coupling to the database: CDC depends on engine-specific features (WAL, binlog)
• Write overhead: Every operation writes an additional row to the outbox table

When to use

• Event-driven systems where event reliability is critical
• Microservices that need to reliably notify other services of changes
• When you implement the Saga Pattern and need to guarantee that the events of each step are published
• Financial or e-commerce systems where losing an event has a business impact
• When you already use a relational database and want to avoid distributed transactions
• Flows where eventual consistency is acceptable but event loss is not

When to avoid

• Systems that require strict real-time event publishing (microseconds)
• When you use a database that does not support ACID transactions (some NoSQL)
• Simple applications where publishing directly to the broker is enough
• Systems with very low event volume where the complexity is not justified
• When you already have Event Sourcing implemented (the event store is already your “outbox”)

Common technologies and implementations

Relationship with other patterns

The Outbox Pattern is a foundational piece in the infrastructure of event-driven systems:

┌─────────────────────────────────────────────────┐
│         Outbox in the ecosystem                  │
│                                                  │
│  ┌──────────┐  needs      ┌──────────────────┐  │
│  │  Saga    │ ──────────► │  Outbox Pattern  │  │
│  │ Pattern  │  to publish │  (this pattern)  │  │
│  └──────────┘  events     └────────┬─────────┘  │
│                                    │             │
│                              at-least-once       │
│                              delivery            │
│                                    │             │
│                                    ▼             │
│                            ┌──────────────┐      │
│                            │ Idempotency  │      │
│                            │   Pattern    │      │
│                            └──────────────┘      │
│                                                  │
│  ┌──────────────┐  uses    ┌──────────────┐     │
│  │ Event-Driven │ ────────►│    Outbox     │     │
│  │ Architecture │  for     │   (reliable)  │     │
│  └──────────────┘  events  └──────────────┘     │
└─────────────────────────────────────────────────┘

• Saga Pattern: Each step of a saga needs to publish events reliably. The Outbox guarantees that no event is lost between steps
• Idempotency Pattern: Since the Outbox guarantees at-least-once, consumers must be idempotent to handle duplicates
• Event-Driven Architecture: The Outbox is the infrastructure that makes event publishing reliable in an event-driven architecture
• Circuit Breaker: If the outbox publisher cannot connect to the broker, the circuit breaker prevents infinite retries
• DDD: The Domain Events generated by each bounded context are published reliably through the outbox

Common mistakes

Next steps

With reliable event publishing solved, the next challenge is protecting your services when dependencies fail. The Circuit Breaker is the pattern that prevents a downed service from dragging down the entire system. Also explore the Idempotency Pattern to ensure that the consumers of your events handle duplicates correctly.