Distributed locks: why leases need fencing tokens

A distributed lock lets only one process at a time do a piece of work across many machines, such as running a scheduled job or updating a shared file. Because the lock holder can crash, these locks are almost always leases that expire on their own, and that expiry creates a gap where two processes both believe they hold the lock. Fencing tokens close the gap: the lock hands out an increasing number with each grant, and the storage you write to rejects any write carrying an older number.

The lock alone can't make your system safe. The protection has to live where the data lives.

What is a lease, and why do distributed locks use one?

A lease is a lock with a time limit. You acquire it for, say, 10 seconds, and if you don't renew it before then, the lock service treats it as released and can give it to someone else.

Leases exist because the alternative is worse. A plain lock that never expires stays held forever if its owner crashes, loses its network or gets terminated mid-job. Every other worker waits on a process that will never come back. A time limit turns a crash into a short delay instead of a permanent stall.

The usual Redis version of a lease looks like this:

SET lock:report-job worker-7f3a NX PX 10000

NX means "only set it if the key does not exist", and PX 10000 sets a 10,000 ms expiry. The value is a random ID unique to this worker, so that when it releases the lock it can check it is deleting its own lock and not one a newer holder acquired. That check-and-delete needs to be atomic, which is why releases are usually done in a small server-side script rather than a separate GET and DEL.

Why can two processes hold the same lock?

The lease protects the lock service from a dead client. It does nothing to stop a client that is slow but still alive.

Picture this sequence:

1. Worker A acquires the lease with a 10 second TTL.
2. Worker A stops running for 15 seconds. A long garbage collection pause, a VM being live-migrated, a laptop lid closing or a process stuck on a swapped-out page can all do this.
3. At 10 seconds the lease expires. Worker B acquires it and starts writing.
4. Worker A wakes up. From its point of view, no time has passed. It still believes it holds the lock and writes its result.

A's write lands after B's, and B's work is silently overwritten. Nothing crashed, no error was logged, and both workers followed the protocol.

You might think A could check the lease before writing. It can, but the pause can happen between the check and the write, so the check proves nothing. Clocks don't help either: A's local clock may also have jumped, or it may simply not have run the code that compares times. Any client-side check has this window. The fix has to be enforced by something that is not paused.

How do fencing tokens work?

A fencing token is a number the lock service returns with every successful acquire, increasing by at least one each time. A gets token 33. When A's lease expires and B acquires, B gets token 34.

Every write to the protected resource includes the token. The resource remembers the highest token it has seen and rejects anything lower. When A wakes up and sends a write with token 33, the storage has already seen 34 and refuses it.

A small simulation shows the difference:

class LockService:
    def __init__(self):
        self.holder, self.expires, self.token = None, 0, 0
    def acquire(self, client, now, ttl):
        if self.holder is None or now >= self.expires:
            self.token += 1
            self.holder, self.expires = client, now + ttl
            return self.token
        return None

class Storage:
    def __init__(self, fenced):
        self.fenced, self.value, self.max_token = fenced, None, 0
    def write(self, value, token):
        if self.fenced and token < self.max_token:
            return f"rejected {value} (token {token} < {self.max_token})"
        self.max_token = max(self.max_token, token)
        self.value = value
        return f"accepted {value} (token {token})"

for fenced in (False, True):
    lock, db = LockService(), Storage(fenced)
    t_a = lock.acquire("A", now=0, ttl=10)   # A gets the lease
    t_b = lock.acquire("B", now=12, ttl=10)  # A paused; lease expired at 10
    print("fenced" if fenced else "unfenced")
    print(" ", db.write("B-data", t_b))
    print(" ", db.write("A-data", t_a))      # A wakes up and writes
    print("  final:", db.value)

Without fencing, both writes are accepted and the final value is A's stale data. With fencing, A's write is rejected with token 1 < 2 and B's data survives. The lock service behaved identically in both runs. Only the storage changed.

How do you enforce a fencing token in a database?

If the protected resource is a row in a SQL database, the check can be a condition on the update itself. Store the last accepted token in a column and only write when the incoming token is at least as high:

UPDATE jobs
SET result = $1, fence = $2
WHERE id = $3
  AND fence <= $2;

If the update affects zero rows, your token is stale and you should stop work rather than retry. Using <= lets the current holder write more than once with the same token; a strictly lower token always fails. Because the comparison and the write happen in one statement, there is no window for a pause to slip into.

The same idea works anywhere you can do a conditional write: object stores with conditional requests, a version column used for optimistic concurrency, or an API that accepts an expected version. If the storage has no way to compare and reject, a fencing token has nowhere to be checked, and the lock is only a strong hint.

Where do the tokens come from?

The token has to be monotonic, meaning it never goes backwards, even if the lock service restarts or fails over. That rules out a counter kept only in one node's memory.

Coordination systems built on consensus already have such numbers. etcd assigns every change a revision that only increases across the cluster, and ZooKeeper does the same with its transaction IDs. Either can serve as a fencing token: use the revision of the write that created your lock key. A single Redis instance can generate a counter with INCR, but a counter that can be lost or rolled back on failover can hand out a number that was already used, so check what guarantees your setup gives before relying on it.

When do you need fencing at all?

Not every lock protects correctness. It helps to ask what happens if two holders overlap for a few seconds.

• Efficiency locks stop duplicate work, like two workers sending the same daily report. Overlap costs some wasted compute or a duplicate email. A lease alone is usually fine, and making the work idempotent covers the rest.
• Correctness locks protect data, like only one writer updating an account balance or a file. Overlap corrupts state. These need fencing, or need the database's own transactions to do the job instead.

Often the cleanest answer is to skip the external lock entirely and let the database serialize the work with a transaction, a unique constraint or a conditional update. A related approach is the transactional outbox pattern, which avoids coordinating two systems by keeping the critical write in one place.

Key takeaways

• Distributed locks are leases, and leases can expire while their holder is still running.
• A paused process can't detect that its lease expired, so client-side checks don't make it safe.
• Fencing tokens are increasing numbers issued with each lease; storage rejects lower ones.
• Enforce the token in the same operation as the write, such as a conditional SQL update.
• Use consensus-backed counters like etcd revisions or ZooKeeper transaction IDs as tokens.

FAQ

Can I just make the lease TTL longer?

A longer TTL makes overlap less likely, but pauses have no upper bound, so it never makes it impossible. It also means a crashed holder blocks everyone else for longer.

What should a worker do when its write is rejected?

Stop. A rejected token means another worker now owns the job, so the right move is to abandon the current work rather than retry with the same token.

Does renewing the lease fix the problem?

Renewal helps a healthy process keep its lease during long work, but a process paused for longer than the TTL can't renew. Fencing still has to catch the write when it wakes up.