Exponential backoff with jitter: retries that don't cause outages

Exponential backoff with jitter is a retry strategy where each retry waits roughly twice as long as the last, plus a random amount, so failed clients spread out instead of hitting a struggling service in lockstep. Backoff gives the service room to recover; jitter stops a thousand clients from retrying at the same millisecond. Add a cap on attempts and only retry errors that can succeed on a second try, and retries stop making outages worse.

Retries look like pure upside. A request failed, you try again, it works. The trouble starts when every client does the same thing at the same time, against a service that is already overloaded.

Why do naive retries make outages worse?

A retry storm is a surge of repeated requests from clients retrying failures, which adds load to a service at the exact moment it has the least capacity. If a service is failing because it is overloaded, every retry is more of the load that caused the failure.

Three things make it worse:

• Immediate retries. A loop that retries with no delay can multiply traffic several times over within milliseconds.
• Synchronised clients. When a dependency blips, every client sees the failure at the same moment. If they all wait the same fixed delay, they all come back at the same moment too.
• Retries at every layer. If a browser retries an API, the API retries a service, and the service retries the database, the attempts multiply. Three layers that each make up to 4 attempts can turn one user action into 4 × 4 × 4 = 64 calls at the bottom.

That last one is the easiest to miss, because each layer looks reasonable on its own.

How does exponential backoff work?

Exponential backoff means the wait before retry n grows geometrically, usually base × 2^n, up to a maximum called the cap. With a 100 ms base, the waits go 100 ms, 200 ms, 400 ms, 800 ms, 1.6 s, 3.2 s.

The early retries are quick, so a brief blip costs the user almost nothing. The later ones back off hard, so a real outage is not hammered by the same client every 100 ms. The cap keeps a single request from waiting minutes.

Backoff alone does not fix synchronisation, though. If 1,000 clients fail at time zero and all use the same schedule, they all retry at exactly 100 ms, then all at 300 ms, then all at 700 ms. You have spread the load over time, but in sharp spikes rather than evenly.

What is jitter, and which kind should you use?

Jitter is randomness added to the delay so that clients that failed together retry at different times. There are three common variants.

To see the difference, here is a simple simulation: 1,000 clients fail at the same moment and each retries up to six times with a 100 ms base and a 10 s cap. The number that matters is the worst 10 ms window, which is roughly how much the recovering service gets hit at once.

import random, collections

BASE, CAP, CLIENTS, ATTEMPTS = 0.1, 10, 1000, 6

def full_jitter(n, prev):
    return random.uniform(0, min(CAP, BASE * 2 ** n))

def peak(delay_fn):
    buckets = collections.Counter()
    for _ in range(CLIENTS):
        t, prev = 0, BASE
        for n in range(ATTEMPTS):
            prev = delay_fn(n, prev)
            t += prev
            buckets[int(t * 100)] += 1   # 10 ms buckets
    return max(buckets.values())

The results:

• No jitter: all 1,000 retries land in the same 10 ms window, every round.
• Equal jitter: around 200 in the worst window.
• Full jitter: around 160.
• Decorrelated jitter: around 70.

The exact numbers move with the random seed, but the shape does not. Any jitter is a large improvement over none. Full jitter is the usual default because it is one line and easy to reason about. Equal jitter is useful when you want every retry to wait at least some minimum. The same idea of spreading out synchronised work shows up in cache stampede prevention, where jittered TTLs stop many keys from expiring at once.

Which errors should you retry?

Only retry when a second attempt has a real chance of succeeding. A transient error is one caused by a temporary condition, like a timeout, a dropped connection, or a briefly overloaded server. A permanent error will fail the same way every time.

A reasonable starting rule for HTTP:

• Retry: network errors, timeouts, 502, 503, 504, and 429 Too Many Requests.
• Do not retry: 400, 401, 403, 404, 422, and most other 4xx responses. The request itself is wrong.
• Respect Retry-After. When a 429 or 503 response includes this header, it tells you how long to wait. Use it instead of your own delay. If you control the server, sending it is the polite way to pair a rate limiter with well-behaved clients.

Here is a small helper with full jitter and an error filter:

const sleep = ms => new Promise(r => setTimeout(r, ms));

function isRetryable(err) {
  if (err.status === 429 || err.status === 503) return true;
  if (err.status >= 400 && err.status < 500) return false;
  return true; // network errors, timeouts, other 5xx
}

async function withRetry(fn, { attempts = 4, base = 100, cap = 5000 } = {}) {
  for (let i = 0; ; i++) {
    try {
      return await fn();
    } catch (err) {
      if (i + 1 >= attempts || !isRetryable(err)) throw err;
      const ceiling = Math.min(cap, base * 2 ** i);
      await sleep(Math.random() * ceiling); // full jitter
    }
  }
}

A function that throws a 503 twice and then succeeds returns on the third call. One that throws a 400 gives up after the first call. One that keeps throwing 500 gives up after four attempts.

Retries need idempotency

Retrying is only safe if doing the operation twice has the same effect as doing it once. That property is called idempotency. Reads are naturally idempotent. A PUT that sets a value is too. A POST that charges a card or creates an order is not.

The tricky case is a timeout. The client gave up waiting, but the server may have finished the work. A blind retry then charges the card twice. The standard fix is an idempotency key: the client generates a unique ID per logical operation, sends it with every attempt, and the server stores the result under that key. A repeat with the same key returns the stored result instead of doing the work again.

If an endpoint cannot be made idempotent, do not retry it automatically.

Limit retries across the whole system

Per-request limits are not enough during a real outage. If every request is allowed three retries and every request is failing, you still triple the load. Two controls help:

• Retry budget. Allow retries only up to a fraction of normal traffic, for example retries may add at most 10% on top of first attempts. Once the budget is spent, fail fast.
• Retry at one layer. Pick the layer closest to the failure, usually the client of the flaky dependency, and have the layers above pass errors through instead of retrying again.

A circuit breaker goes further: after enough consecutive failures it stops calling the dependency entirely for a short period, then lets a few test requests through to see if it has recovered. It pairs well with backoff, since backoff handles blips and the breaker handles sustained outages.

Key takeaways

• Retries without delay or randomness add load to a failing service exactly when it can least handle it.
• Exponential backoff spaces out retries; jitter stops clients from retrying in lockstep. You need both.
• Full jitter is a sensible default and takes one line of code.
• Only retry transient errors, honour Retry-After, and make retried operations idempotent.
• Cap retries per request and across the system, and retry at one layer, not every layer.

FAQ

How many times should you retry a failed request?

Usually two or three retries for an interactive request. Beyond that, the user is waiting too long and the extra attempts mostly add load. Background jobs can retry more, with a larger cap.

Is full jitter or decorrelated jitter better?

Both spread retries well. Full jitter is simpler and bounded by the attempt number; decorrelated jitter spreads slightly more in simulations but is harder to reason about. Either is far better than no jitter.

Should I retry on a 500 error?

Often yes, a small number of times, since many 500s are transient. But if the same request keeps returning 500, the cause is likely a bug, and retrying will not fix it.