Transactional outbox pattern: reliable events without dual writes

The transactional outbox pattern solves the dual write problem: saving a change to your database and publishing an event about it, without one happening while the other is lost. Instead of writing to the database and then sending to a message broker, you write the business row and an event row to an outbox table in the same transaction. A separate relay process reads that table and publishes the events, so every committed change produces its event at least once.

It is a small idea with a lot of reach. Most event-driven bugs that show up as "the email never went out" or "search is missing this order" trace back to a write that reached one system and not the other.

What is the dual write problem?

A dual write is any operation that has to update two separate systems, such as a database and a message broker, with no shared transaction between them. Each write can succeed or fail independently, so there is always a window where they disagree.

The usual code looks harmless:

await db.query("INSERT INTO orders (id, total) VALUES ($1, $2)", [id, total]);
await broker.publish("order.created", { id, total });

Now walk through the failure cases:

• The insert commits, then the process crashes before publish runs. The order exists, but nothing downstream ever hears about it.
• The broker is down or times out. You can retry, but if the process restarts during the retry loop, the event is gone.
• You swap the order and publish first. Now a consumer can receive an event for an order that later fails to commit, because of a constraint violation or a rollback.

Wrapping both calls in a database transaction does not help. The broker is not part of that transaction, so a rollback cannot take back a message that was already sent. Two-phase commit, a protocol that coordinates a single commit across several systems, could in theory, but most brokers do not support it and it adds latency and a coordinator that can itself fail.

How does the transactional outbox pattern work?

The trick is to turn two writes into one. The event becomes a row in the same database as the business data, written in the same transaction. Either both rows commit, or neither does.

CREATE TABLE outbox (
  id           UUID PRIMARY KEY,
  topic        TEXT NOT NULL,
  payload      JSONB NOT NULL,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at TIMESTAMPTZ
);

Writing an order then looks like this:

BEGIN;
INSERT INTO orders (id, total) VALUES ('o_123', 4200);
INSERT INTO outbox (id, topic, payload)
VALUES (gen_random_uuid(), 'order.created', '{"order_id":"o_123","total":4200}');
COMMIT;

The second half is the relay, a background process that reads unpublished outbox rows, sends them to the broker, and marks them as sent. It runs on its own schedule, so a slow or unavailable broker no longer blocks the request that created the order. The event simply waits in the table until the broker is reachable again.

Here is the whole flow as a small Python script against SQLite, with a simulated crash in the middle of the second order:

def place_order(total, fail=False):
    order_id = str(uuid.uuid4())
    with db:  # one transaction
        db.execute("INSERT INTO orders VALUES (?, ?)", (order_id, total))
        db.execute("INSERT INTO outbox (id, topic, payload) VALUES (?, ?, ?)",
                   (str(uuid.uuid4()), "order.created",
                    json.dumps({"order_id": order_id, "total": total})))
        if fail:
            raise RuntimeError("crash before commit")

def relay(publish, batch=100):
    rows = db.execute("SELECT id, topic, payload FROM outbox "
                      "WHERE published_at IS NULL ORDER BY created_at LIMIT ?",
                      (batch,)).fetchall()
    for msg_id, topic, payload in rows:
        publish(topic, msg_id, payload)
        with db:
            db.execute("UPDATE outbox SET published_at = CURRENT_TIMESTAMP "
                       "WHERE id = ?", (msg_id,))
    return len(rows)

After one successful order and one that crashes before commit, there is one row in orders and one in outbox. The crashed order left nothing behind in either table. Running the relay publishes one message; running it again publishes zero, because the row is already marked.

Why do consumers still need to handle duplicates?

Look closely at the relay. It publishes, then marks the row. If the process dies between those two steps, the next run publishes the same event again. The outbox gives you at-least-once delivery: every event is sent one or more times, never zero.

Exactly-once delivery across a database and a broker is not something the pattern promises. What you can get is exactly-once effect, by making consumers idempotent. The simplest way is to send the outbox row's id as a message ID and have each consumer record which IDs it has already handled, in the same transaction as its own side effects:

BEGIN;
INSERT INTO processed_messages (message_id) VALUES ($1)
ON CONFLICT DO NOTHING;
-- if zero rows were inserted, this is a duplicate: commit and stop
-- otherwise, apply the side effect here
COMMIT;

The first delivery inserts the ID and does the work. A redelivery hits the primary key, inserts nothing, and skips. This is the same idempotency idea that makes retries with exponential backoff safe, applied on the receiving side.

Polling relay or change data capture?

There are two common ways to get rows out of the outbox table.

Change data capture (CDC) means reading the database's own change log and turning each committed row change into an event. With CDC, the relay does not need a published_at column at all, because the log position tracks progress. Many teams insert outbox rows and delete them shortly after, since the log already captured them.

Polling is the right starting point for most systems. If you run more than one polling relay for availability, have each one claim rows with SELECT ... FOR UPDATE SKIP LOCKED in PostgreSQL, so two relays never pick up the same batch at once.

When should you use the outbox pattern?

Use it when a state change and the event describing it must not drift apart:

• An order, payment or signup must trigger downstream work like emails, invoices or fulfilment.
• Other services keep read models or search indexes built from your events.
• You are splitting a monolith and need services to learn about each other's changes reliably.

It is overkill for fire-and-forget signals where losing an occasional event is fine, such as analytics pings or cache warm-up hints. It also does not help if the data and the event live in different databases; the outbox has to sit next to the rows it describes.

A few operational details matter in practice:

• Clean up. Delete or archive published rows on a schedule, or the table grows forever and polling slows down.
• Watch the lag. Alert on the age of the oldest unpublished row. A growing number means the relay is stuck or the broker is down.
• Ordering. A single relay reading by created_at keeps rough order, but parallel relays do not. If per-entity order matters, partition by entity ID so that all events for one order go through the same lane.

Key takeaways

• Writing to a database and then a broker is a dual write, and it will eventually lose or invent events.
• The outbox pattern writes the event into the same database transaction as the data, so both commit or neither does.
• A relay publishes outbox rows asynchronously, by polling or by change data capture.
• Delivery is at-least-once, so consumers must deduplicate by message ID.
• Monitor outbox lag and clean up published rows.

FAQ

Is the transactional outbox pattern the same as event sourcing?

No. Event sourcing stores events as the primary record and derives state from them. The outbox keeps your normal tables as the source of truth and uses events only to notify other systems.

Does the outbox pattern add latency to requests?

Barely. The request does one extra insert inside a transaction it was already running. Event delivery is delayed by the relay's poll interval or CDC lag, usually well under a second.

Can I use the outbox pattern with a NoSQL database?

Yes, if the database can write the document and the event atomically, for example in a single document or a multi-document transaction. Some databases also expose change streams that can act as the relay.