Reliability
An acknowledgement means your order is on disk. This page is what that acknowledgement survives, failure by failure, and what each tier promises about it.
Every command that reaches the engine is appended to a journal and fsynced before the engine acts on it. The answer you receive already reflects durable state: there is no window in which an acknowledged order exists only in memory, and there is no asynchronous write-behind whose loss would need explaining later. That is the root fact of this page, and everything below is arithmetic on top of it.
The second fact is that determinism is the recovery mechanism, not a slogan. The engine is a pure, single-writer state machine over an ordered log, so replaying the same log reproduces the same state, byte for byte, verified by a state hash. Every recovery path below is therefore the same operation, a replay, and it is the operation that runs in every test, every restart and every migration. Failover is not special code that waits all year for the worst day; it is the code that runs every day.
The trade we chose, stated plainly
Infrastructure that must survive a dying machine has two honest designs.
Consensus replication. A quorum of nodes agrees on every command before it is acknowledged. This buys failover measured in milliseconds, and it costs two things: a quorum round trip inside every acknowledgement, and a durability story that lives in the replicas rather than on any disk. Lose the quorum at once, which one datacenter event can do, and there is no volume to pick up.
A durable journal. Every command is fsynced to a volume that outlives the machine, and copies leave the machine continuously. This buys durability that survives a total loss of power, and recovery that does not require any surviving leader: a fresh machine rebuilds from the shipped copy alone. It costs failover measured in seconds to minutes, because recovery is a replay.
Clobber is the second design, and we state its cost instead of hiding it. The rest of this page prices that cost per tier, per failure.
What an acknowledgement survives
Two definitions, then the table. RPO, recovery point objective: how much acknowledged work a failure may cost. RTO, recovery time objective: how long until orders flow again. A number labelled measured traces to a run on the hardware of the tier that claims it; a number labelled target is a design figure we publish as such and contract only once it is measured.
| Failure | What happens | Production | Dedicated |
|---|---|---|---|
| The process dies | restart in place, replay the tail | RPO 0, back in about a minute | RPO 0, standby makes it faster |
| The machine dies | the data volume outlives the instance; a replacement attaches it and replays | RPO 0, minutes | RPO 0; a hot standby in a second datacenter takes over, RTO target 2 minutes |
| The disk or the datacenter dies | journal segments and snapshots ship continuously to object storage that spans datacenters | RPO target 10 seconds, restore runbook under an hour | the standby already holds the state: RPO 0 to 2 seconds, RPO 0 with the synchronous replicas add-on, RTO target 2 minutes |
| The region dies | the shipped copy is replicated to a second region | best-effort restore; replication typically lags under a minute | with the cross-region add-on: a contractual replication ceiling of 15 minutes, typically under a minute, and a restore that is drilled, not described, RTO target 4 hours |
Three details behind the rows:
- The volume is the first replica. Cloud block storage is already replicated inside its datacenter, which is why machine death costs nothing on any tier, and why we do not sell same-datacenter mirrors: that would be selling the cloud's built-in durability twice.
- The shipping is continuous, not nightly. Closed journal segments upload the moment they rotate; the active segment ships in small increments every few seconds, cut on record boundaries so a restore is always a valid journal. The 10 second figure is twice the shipping cadence, as margin.
- A fenced primary cannot corrupt anything. Promotion of a standby is guarded by an atomic epoch lease: a machine that lost its lease can still be running, but its late writes land under a dead epoch and recovery ignores them by construction. Two writers on one log is the one unrecoverable failure in this design, so it is the one made structurally impossible rather than merely unlikely.
What Production buys, and what Dedicated buys
The two tiers run the same engine, the same journal discipline and the same shipping. What differs is what we are willing to put a number on, and that difference is the honest one: every published number is measured on the storage of the tier that claims it, and a tier whose storage does not earn a number does not get one.
Production runs on shared cells with general-purpose volumes. It sells uptime and correctness: a 99.9% monthly SLA, durable acknowledgements, the full ladder above. It does not carry a latency number, because its storage class does not earn one under load, and we would rather publish nothing than a figure taken on quiet hardware. The SLA arithmetic, stated plainly: 99.9% is a 43 minute monthly budget, so a datacenter-scale event will likely consume it, and the credit gets paid. That is the deal at this price, and it is a good one.
Dedicated is one tenant on its own engine, its own machine and its own provisioned-IOPS volume, in your region. That storage earns its numbers: durable acknowledgements in single digit milliseconds at p99 at the guaranteed sustained rate, measured on the tier's own disk, with 3,000 orders per second guaranteed on Base and 5,000 on Premium. It carries the 99.99% SLA, which is a 4.4 minute monthly budget, and the only honest way to promise 4.4 minutes is a hot standby that is already in the same state, which Dedicated has. The disaster-recovery add-ons stack on top: synchronous journal-fed replicas in additional datacenters for RPO 0, and the cross-region contract with its drilled restore.
Recovery is a replay, and the replay is measured
The numbers that make the table above arithmetic rather than hope, all measured on the 2 vCPU cell hardware unless noted:
- Replay runs at about 389,000 commands per second through real matching flow, with the restored state verified against the original by hash. A snapshot is taken at most 10 million commands ago, so the worst-case tail replays in about 26 seconds.
- Snapshots are small and fast: a million resting orders is about 56 MiB, roughly half a second to write and about the same to restore.
- The crash path is tested the unpleasant way: the process is killed mid-load and restarted. Zero acknowledged commands lost, zero duplicated events, invariants intact.
- The ledger inside the engine is checked as it moves: double entry, integer arithmetic, conservation verified continuously. A recovery that reconstructs the wrong balances fails loudly instead of reporting success.
One property here is worth naming because teams that build their own infrastructure keep rediscovering its absence: recovery does not need the failed machine. A standby, or a completely fresh machine, catches up from object storage alone, with no participation from the primary. Systems that replicate state from a live leader inherit the leader as a dependency of their own recovery; a journal in object storage has no such dependency, and that is much of why we chose it.
Your region, and your way out
Dedicated runs in the region you name, and the same discipline extends to clouds: running your environment on another major cloud is an engagement, and the first line item of that engagement is the measurement campaign, the same one that produced every number on this page, run on that cloud's premium storage. The results set the guarantees your contract carries. No number is contracted that was not measured where you run.
The way out is in the contract, not in a promise: a data export guarantee and a wind-down period. The journal is a complete, ordered, replayable record of everything your markets ever did, which means an export is not a database dump that hopes to be complete; it is the same artifact our own recovery uses.
What each claim rests on
In the spirit of the fundamentals page: each claim, and whether it holds by construction, by measurement, or as a stated target.
| Claim | Rests on |
|---|---|
| An acknowledged command is on disk | construction: the fsync precedes the engine, always |
| Replay reproduces state exactly | construction, and a hash check on every recovery and every test |
| RPO 0 on process and machine death | construction: the volume outlives both |
| Single digit ms p99 acks on Dedicated | measured, on the tier's own storage, at the guaranteed rate |
| RPO 10 seconds on datacenter loss (Production) | target: twice the shipping cadence; contracted only once measured |
| RTO 2 minutes on Dedicated | target: replay speed is measured, the drill makes it contractual |
| Cross-region replication ceiling 15 minutes | the object store's own replication SLA, with typical lag under a minute |
| The books balance after any recovery | construction: the ledger is inside the engine and conservation is checked as it moves |