Reliability

An acknowledgement means your order is on disk. This page is what that acknowledgement survives, failure by failure, and what each tier promises about it.

Every command that reaches the engine is appended to a journal and fsynced before the engine acts on it. The answer you receive already reflects durable state: there is no window in which an acknowledged order exists only in memory, and there is no asynchronous write-behind whose loss would need explaining later. That is the root fact of this page, and everything below is arithmetic on top of it.

1 router: TLS, key to tenant, tenant to cell stateless, any router serves any tenant 2 gateway: authenticate, check idempotency reads the clock once, stamps IssuedAt on the command parses decimal strings into scaled integers, once 3 journal: append, fsync record = length + crc32c + payload durable from here 4 engine applies, emits events pure state machine, one goroutine, microseconds 5 the acknowledgement returns it already reflects fills from entry: the engine ran before the response 6 feed fans out per market seq, async 6 projector writes history async, never on the ack path done once, at the edge group commit, no timer w1 append─fsync━━━━━━━┓ w2 append─wait......┃ done w3 append─wait....┃ done w4 append─wait..┃ done one fsync covers the whole batch: the batch is whatever arrived during the in-flight fsync, a lone writer waits zero
Where durability happens in the life of one order. Everything above the amber line can fail and cost nothing: the client never got an acknowledgement. Everything below it is downstream of a record that is already on disk. The inset is why the fsync is affordable at load: concurrent writers share one flush, so the effective commit window is a single fsync latency, set by the storage, with no timer adding delay when the cell is quiet.

The second fact is that determinism is the recovery mechanism, not a slogan. The engine is a pure, single-writer state machine over an ordered log, so replaying the same log reproduces the same state, byte for byte, verified by a state hash. Every recovery path below is therefore the same operation, a replay, and it is the operation that runs in every test, every restart and every migration. Failover is not special code that waits all year for the worst day; it is the code that runs every day.

The trade we chose, stated plainly

Infrastructure that must survive a dying machine has two honest designs.

Consensus replication. A quorum of nodes agrees on every command before it is acknowledged. This buys failover measured in milliseconds, and it costs two things: a quorum round trip inside every acknowledgement, and a durability story that lives in the replicas rather than on any disk. Lose the quorum at once, which one datacenter event can do, and there is no volume to pick up.

A durable journal. Every command is fsynced to a volume that outlives the machine, and copies leave the machine continuously. This buys durability that survives a total loss of power, and recovery that does not require any surviving leader: a fresh machine rebuilds from the shipped copy alone. It costs failover measured in seconds to minutes, because recovery is a replay.

Clobber is the second design, and we state its cost instead of hiding it. The rest of this page prices that cost per tier, per failure.

What an acknowledgement survives

Two definitions, then the table. RPO, recovery point objective: how much acknowledged work a failure may cost. RTO, recovery time objective: how long until orders flow again. A number labelled measured traces to a run on the hardware of the tier that claims it; a number labelled target is a design figure we publish as such and contract only once it is measured.

Failure What happens Production Dedicated
The process dies restart in place, replay the tail RPO 0, back in about a minute RPO 0, standby makes it faster
The machine dies the data volume outlives the instance; a replacement attaches it and replays RPO 0, minutes RPO 0; a hot standby in a second datacenter takes over, RTO target 2 minutes
The disk or the datacenter dies journal segments and snapshots ship continuously to object storage that spans datacenters RPO target 10 seconds, restore runbook under an hour the standby already holds the state: RPO 0 to 2 seconds, RPO 0 with the synchronous replicas add-on, RTO target 2 minutes
The region dies the shipped copy is replicated to a second region best-effort restore; replication typically lags under a minute with the cross-region add-on: a contractual replication ceiling of 15 minutes, typically under a minute, and a restore that is drilled, not described, RTO target 4 hours

Three details behind the rows:

What Production buys, and what Dedicated buys

The two tiers run the same engine, the same journal discipline and the same shipping. What differs is what we are willing to put a number on, and that difference is the honest one: every published number is measured on the storage of the tier that claims it, and a tier whose storage does not earn a number does not get one.

Production runs on shared cells with general-purpose volumes. It sells uptime and correctness: a 99.9% monthly SLA, durable acknowledgements, the full ladder above. It does not carry a latency number, because its storage class does not earn one under load, and we would rather publish nothing than a figure taken on quiet hardware. The SLA arithmetic, stated plainly: 99.9% is a 43 minute monthly budget, so a datacenter-scale event will likely consume it, and the credit gets paid. That is the deal at this price, and it is a good one.

Dedicated is one tenant on its own engine, its own machine and its own provisioned-IOPS volume, in your region. That storage earns its numbers: durable acknowledgements in single digit milliseconds at p99 at the guaranteed sustained rate, measured on the tier's own disk, with 3,000 orders per second guaranteed on Base and 5,000 on Premium. It carries the 99.99% SLA, which is a 4.4 minute monthly budget, and the only honest way to promise 4.4 minutes is a hot standby that is already in the same state, which Dedicated has. The disaster-recovery add-ons stack on top: synchronous journal-fed replicas in additional datacenters for RPO 0, and the cross-region contract with its drilled restore.

Recovery is a replay, and the replay is measured

The numbers that make the table above arithmetic rather than hope, all measured on the 2 vCPU cell hardware unless noted:

the process is killed mid-write len+crc ok len+crc ok len+crc ok torn tail: the crash boundary on start: truncate the torn record, keep everything intact then recovery is arithmetic: newest snapshot crc + ledger hash replay the tail 389,000 commands/s hash-verified against the original serving snapshots at most 10 million commands apart, so the worst-case tail replays in about 26 seconds the case that is not a crash: ok crc fails in the middle ok this is a bad disk, not a crash: replay refuses outright, because coming back with somebody's orders missing and nobody told is the worse failure
The two shapes of a damaged journal, and why they are treated oppositely. A torn record at the tail is what a crash looks like, so it is truncated and the log continues from the last intact record. A record that fails its checksum in the middle of a segment cannot be a crash, so recovery stops instead of guessing, and the standby or the shipped copy takes over. The replay rate is measured on the 2 vCPU cell hardware.

One property here is worth naming because teams that build their own infrastructure keep rediscovering its absence: recovery does not need the failed machine. A standby, or a completely fresh machine, catches up from object storage alone, with no participation from the primary. Systems that replicate state from a live leader inherit the leader as a dependency of their own recovery; a journal in object storage has no such dependency, and that is much of why we chose it.

Your region, and your way out

Dedicated runs in the region you name, and the same discipline extends to clouds: running your environment on another major cloud is an engagement, and the first line item of that engagement is the measurement campaign, the same one that produced every number on this page, run on that cloud's premium storage. The results set the guarantees your contract carries. No number is contracted that was not measured where you run.

The way out is in the contract, not in a promise: a data export guarantee and a wind-down period. The journal is a complete, ordered, replayable record of everything your markets ever did, which means an export is not a database dump that hopes to be complete; it is the same artifact our own recovery uses.

What each claim rests on

In the spirit of the fundamentals page: each claim, and whether it holds by construction, by measurement, or as a stated target.

Claim Rests on
An acknowledged command is on disk construction: the fsync precedes the engine, always
Replay reproduces state exactly construction, and a hash check on every recovery and every test
RPO 0 on process and machine death construction: the volume outlives both
Single digit ms p99 acks on Dedicated measured, on the tier's own storage, at the guaranteed rate
RPO 10 seconds on datacenter loss (Production) target: twice the shipping cadence; contracted only once measured
RTO 2 minutes on Dedicated target: replay speed is measured, the drill makes it contractual
Cross-region replication ceiling 15 minutes the object store's own replication SLA, with typical lag under a minute
The books balance after any recovery construction: the ledger is inside the engine and conservation is checked as it moves