The Complete Responsive CMS Blog created by Francesco Malagrino

A practical deep dive into an orchestration-based saga with timeouts, DLQs, distributed locks, and reverse-order compensation.

Distributed transactions are easy to explain and painful to implement well.

So I built a reference project that shows what a saga looks like beyond slides:
state management, idempotency, retries, timeouts, dead-letter queues, and compensating transactions.

Repo: https://github.com/Vegetam/Saga-pattern-architecture

First: this is orchestration, not choreography

Saga coordination comes in two flavors:

• Choreography: services react to events and the workflow emerges


• Orchestration: a central orchestrator coordinates steps and tracks state

This project is orchestration by design:

• the README calls it “Saga Orchestration Pattern”


• ADR-001 explicitly chooses orchestration over choreography

That choice is deliberate: orchestration gives you one place to see saga progress, enforce timeouts, and run compensations.

What this project implements

At a high level:

• a dedicated Saga Orchestrator Service with a state machine and a compensation manager


• Kafka topics for commands/replies/events (plus DLQs)


• an idempotency strategy to prevent double-processing


• reverse-order compensation logic (rollback)

Step definitions: forward + compensation

The saga is modeled as a list of steps, each with:

• a command (do the work)


• a compensation command (undo the work)


• a timeout

Here’s the shape of the Order saga definition in the repo (formatted):

export const orderSagaDefinition = {
  name: "OrderSaga",
  steps: [
    {
      name: "Reserve Order",
      command: "RESERVE_ORDER",
      topic: "saga-order-commands",
      compensationCommand: "CANCEL_ORDER",
      compensationTopic: "saga-order-commands",
      timeoutMs: 30_000,
    },
    {
      name: "Process Payment",
      command: "PROCESS_PAYMENT",
      topic: "saga-payment-commands",
      compensationCommand: "REFUND_PAYMENT",
      compensationTopic: "saga-payment-commands",
      timeoutMs: 30_000,
    },
    {
      name: "Reserve Inventory",
      command: "RESERVE_INVENTORY",
      topic: "saga-inventory-commands",
      compensationCommand: "RELEASE_INVENTORY",
      compensationTopic: "saga-inventory-commands",
      timeoutMs: 30_000,
    },
  ],
  timeoutMs: 120_000,
};

(See saga-orchestrator/src/sagas/order.saga.ts.)

The non-negotiable: idempotency keys

Kafka is commonly “at least once.” That means the same command can be delivered twice.

This repo protects every step with an idempotencyKey and has services store processed keys so they can safely return the previous result instead of re-running logic.

Key format (from ADR-004):

saga:{sagaId}:step:{stepIndex}  // forward step
saga:{sagaId}:comp:{stepIndex}  // compensation step

This prevents nightmare scenarios like charging a card twice.

Rollbacks: reverse-order compensation

When a step fails, the orchestrator compensates previously completed steps in reverse order. That’s the classic saga rollback rule.

The repo’s CompensationManager dispatches compensation commands backwards and marks the saga COMPENSATION_FAILED if rollback itself fails (which usually means manual intervention is required).

That last point matters: compensation is itself an eventually consistent workflow — it can fail, and you need a plan when it does.

Why orchestration can be worth it

Choreography can be great — but orchestration often wins when:

• workflows have many steps/branches


• you want a single place to query saga state


• you want centralized timeouts/retries/compensation logic


• debugging across many services is becoming painful

That’s the niche this project targets: a clear, explicit “source of truth” for saga state.

What I’d build next

If I took this further toward production-hardening, I’d focus on:

• an Outbox pattern on participant services (stronger guarantees around DB write + message publish)


• schema versioning for commands/replies


• automated DLQ replay tooling + runbooks