Lesson 07 · One truth, many hands

One truth. More places to read it.

Everything so far assumed the machine keeps running. Replication keeps live copies of the same data on other machines — which is not the same thing as a backup, and brings a question of its own: when two machines hold the same row, which one is right?

WALReplica lagRead your writesFailover

01 · The assumption so far

Every technique so far assumed the machine keeps running.

Indexes, partitions, shards — every one of them made a working database do more. None of them considered what happens when it stops working.

One machine holding your data is one machine that can fail. And when it does, the data is not slow. It is unreachable, until someone restores it from somewhere else.

If you sharded, this got worse rather than better. Four keeps are four things that can fail, and losing one does not cost you a quarter of your speed — it costs a quarter of your answers. Every customer whose rows lived there gets errors, and the other three machines cannot help them.

Capacity and survival are different problems. Everything up to now solved the first one. Nothing has touched the second.

02 · What replication is

Continuously updated copies of the same data.

Replication keeps continuously updated copies of the same data on other machines.

One machine — the primary — accepts writes. One or more replicas receive every change and apply it to their own copy, staying seconds or milliseconds behind. Each replica is a complete, working database, not an archive.

Notice what is different from every previous tool. Indexes, partitions, and shards all divide data. Replication duplicates it. Sharding gave four machines a quarter each; replication gives four machines the same thing, four times.

How the copy travels

You already have the mechanism. Lesson 04 followed a write into the write-ahead log — the ordered record of every change, written so a crash can be recovered by replaying it.

Replication is that same log, replayed somewhere else.

Primary:  write  →  WAL  →  commit
                      │
                      └──── stream ────►  Replica:  replay  →  same state

Nothing new was invented. The log that made your data durable now makes copies of it possible.

It travels in one of two forms. Physical replication ships the log as raw bytes, producing an exact copy of the whole cluster — simple and fast, but all-or-nothing. Logical replication decodes it into row-level changes first, which costs more but can be selective per table, work across versions, and be consumed by systems that are not PostgreSQL at all — which is what change-data-capture pipelines and search-index syncing actually are.

03 · Isn’t that just a backup?

They fail in opposite directions.

Both keep a second copy. So why not simply restore last night’s backup when a machine dies?

Because the difference is not freshness or size. It is what each one is for.

A backup is a moment. A replica is a continuous state.

Your backup is 03:00 this morning. Your replica is 200 milliseconds ago.

Recovery times are not comparable

Restoring a large database means pulling it from cold storage, writing it out, replaying archived log files, and rebuilding — hours, during which you are down. Promoting a replica means telling a machine that is already running, already warm, and already holding the data to start accepting writes. Seconds.

A replica does work; a backup does nothing

This part is underrated. A replica is a live database you can read from, so it adds capacity every day, not only on the worst day. A backup is inert bytes you pay to store and hope never to use.

And now the one that settles it

Replication is faithful. That is the entire point of it — and it is also the problem:

DROP TABLE orders;

That statement reaches every replica in milliseconds. So does a bad migration, a buggy UPDATE with a missing WHERE, and an application that quietly corrupts rows for a week. Replicas do not protect you from any of it, because they are doing precisely their job.

A backup from before the mistake is the only thing that saves you.

Replication protects you from a machine failing. Backups protect you from yourself.

Two more consequences follow. Backups hold history; replicas hold only now. "Restore this table to how it looked last Tuesday" is a backup question — a replica has been faithfully overwriting last Tuesday ever since. And backups can be made unreachable: stored offsite, in another account, immutable. A replica is by definition connected to the primary, so whatever reached your primary can reach it too.

So the answer is that they answer different questions, and you need both.

04 · Where it pays

Six things a second copy can do.

  • Surviving a machine failure. A replica is promoted and the system continues. This is the reason it exists.
  • Read capacity. Every replica is another CPU and another buffer pool answering queries. For read-heavy applications — most applications — this is the cheapest scaling move available.
  • Maintenance without downtime. Upgrade a replica, promote it, then rebuild the old primary. The machinery that survives an accident also removes planned outages.
  • Reads closer to users. A replica in another region answers local reads without a transatlantic round trip.
  • Feeding other systems. Logical replication keeps a search index, a warehouse, or an event stream current without anyone polling your primary.
  • Keeping analytics off the primary. Long reporting queries run somewhere they cannot evict the pages your transactional traffic depends on.

05 · What it does not fix

More copies, same primary.

Write throughput. Every write still goes through the single primary — and now must additionally be shipped and replayed N times. Replicas make writes marginally worse. If writes are your bottleneck, replication is not the answer — sharding is, and that was the previous lesson.

Your working set. Every replica holds the full dataset, so each has an identical memory problem. If your working set is 500 GB and your machines have 128 GB of RAM, ten replicas give you ten machines thrashing equally. Replication multiplies read throughput, never memory efficiency.

And it introduces a failure with no clean answer

When the primary dies and a replica is promoted, the old primary may come back — and start accepting writes too. Two machines, both convinced they are authoritative, both committing. That is split-brain: two divergent histories, both "committed", frequently impossible to reconcile. You pick a winner and lose the other side’s work. Preventing it needs fencing or a consensus protocol, not a health check that noticed a timeout.

So when two machines hold the same row, which one is right?

While the primary is alive, the answer is simple: the primary is, by definition. It is the only machine permitted to accept writes, so it holds the newest version of everything. A replica is behind it, never ahead of it. Which copy is authoritative is not a puzzle — it is a rule, and the rule is the same every time.

The question only gets hard in two moments. When you read from a replica, you may get an answer that was true a moment ago rather than one that is true now — and only your product knows whether that is acceptable for that particular read. And when the primary dies, the rule has nothing left to point at: some replica must be declared right, and whatever it had not yet received is simply gone.

Both are the same underlying question, and it is not really a database question: what promise are you making about freshness, and to whom? Answering that deliberately, per operation rather than per system, is the last lesson.

The Old Dragon “A copy is useful only when the vault agrees which page is newest—and which scribe may still write.”

The Dragon’s Reckoning

Where we stand

Reads can spread across machines. Writes still pass through one authority, and every copy may trail it.

What we learnedReplicas replay WAL

Each copy adds CPU, RAM, and storage for reads while following the primary’s ordered changes.

What we solvedRead-heavy pressure

Suitable queries can leave the primary and use replica capacity.

What remainsOne full data set per machine

Copies neither shrink the working set nor raise the primary’s write ceiling.

Our system nowPrimary · replicas · routing

We added lag policy, acknowledgement choices, and a real failover procedure.

Next · Lesson 08

The Price of Many Realms

Choose consistency per operation and finish with the full scaling decision matrix.

The source document turns next to partitioning.