Determinism as a product decision

How Clobber builds a hosted central limit order book, and why the boring choices are the ones that matter.

Clobber is a matching engine you rent instead of build. This is a description of how it works and why it is built this way, written for people who are considering building one themselves, because that is the only audience that can judge whether the answers here are good ones.

It is not a sales document. Every engineering decision it argues for is visible in the published API, which is the point: the contract is written first and the implementation is measured against it.

Several of the claims below are theorems rather than opinions, and a few are empirical laws with fitted parameters. Those are stated formally, with proofs and measurements, in the companion document: Fundamentals. Where the two disagree, the companion is right and says so explicitly.

The problem worth solving

Building a prediction market, a derivatives venue or an exchange means building two products. One is the thing you set out to make: the markets, the audience, the reason anyone shows up. The other is an order book with a ledger under it, and it is the one that eats the calendar.

The pattern is easy to observe. Teams reach for the best available open source matching engine, usually a Java library built on the LMAX Disruptor, and discover that matching was the part that was already solved. What is missing is everything around it: durability that survives a power cut, a gateway with authentication and idempotency, a realtime feed, settlement, failover, and the ability to prove to a regulator or a customer what happened and in what order. Three months becomes a year. The venue that was going to differentiate on markets differentiates on having survived its own infrastructure.

Clobber's bet is that this layer is the same for everybody and should be bought.

One decision holds the rest up

The matching engine is a pure state machine. It performs no IO, reads no clock, uses no randomness, holds no floating point number and runs on one thread. Every input arrives as a command from a totally ordered log. Replaying the same log produces byte identical state and the same events, in the same order.

That is a constraint, and it is inconvenient in small ways every day. It is worth it because it is the same answer to four different questions.

How do you fail over? You replay. A standby that has consumed the same log is in the same state by construction, not by reconciliation. There is no divergence to detect because there is no way for divergence to occur.

How do you prove what happened? You replay. The log is the record and the state is a function of it, so an audit is a computation rather than an argument. Any state the system was ever in can be reconstructed exactly.

How do you move a customer between machines? You replay. Halt, snapshot, transfer, replay on the target, compare state hashes. If the hashes match, the target is the source. If they do not, nothing was moved, because the comparison happens before anything is switched over.

How do you know the engine is correct? You replay. Feed the same commands to two engines and compare. Feed them to a deliberately naive second implementation and compare. Feed them, restart the process halfway, and compare. Determinism turns testing from sampling into equality.

Four hard problems answered by one property is why the property is defended above everything else.

Money is integers, and rounding happens once

There are no floating point numbers anywhere in the engine. Prices, quantities and balances are 64 bit integers in scaled units, and the API speaks decimal strings, converted exactly once at the edge.

More interesting than the choice of integers is what the engine does with them. When a market is created it checks that its tick size multiplied by its lot size lands exactly on the settlement currency's smallest unit. A market that fails this check is rejected at creation. The result is that no fill, no hold and no settlement in that market can ever round, because every value that can be expressed is exact by construction.

That leaves exactly one place in the system where a number is rounded: fees, which are a percentage and cannot avoid it. The rounding is half away from zero and symmetric, so a rebate rounds the same way a charge does and the venue never profits from the rounding rule itself.

One rounding site, chosen deliberately, is the difference between a ledger you can reason about and one you audit.

The ledger is inside the engine, and it proves itself

Balances are not a table the engine writes to. They are engine state, double entry, and the counterparty to every user account is a system account: a funding account that mirrors what the venue asserts exists outside, an escrow account per market holding collateral, a fee account.

Because every entry has both sides inside the same state machine, the sum of all balances of any asset within a tenant is exactly zero, at every instant. That is checkable, so it is checked. In development builds the engine verifies it after every single command; in production after every snapshot, which bounds how far a bad state could travel without putting an O(n) walk on the hot path.

When the check fails, the process stops. It does not log and continue. Value was created or destroyed, and every entry written after that point would be arithmetically correct on top of a false premise. Stopping and letting a standby replay the log is strictly better than a ledger that looks fine and is not. This is the same argument as fsync: the expensive behaviour is the one that lets you make a promise.

Positions live in this ledger as assets rather than in a table of their own, which makes settlement an ordinary double entry transaction instead of a special case, and lets one conservation rule cover money and contracts together.

Fully collateralised markets, and the rule the tests corrected

A binary contract settles at one or zero. A buyer pays the price; a short seller posts the rest. Together they escrow exactly the full value of the contract, so settlement pays winners out of money that was always there. There is no credit, no margin call, and no scenario where the venue owes more than it holds.

The interesting part is a correction that a test made to the design.

The obvious statement of the rule is per trade: a buyer escrows the price, a seller escrows one minus the price. A property test found on its second generated trade that this is wrong. It is right only when both sides are starting from flat. When a trade closes an existing short position, applying the per trade rule takes collateral for exposure that no longer exists, and escrow ends up holding more than the market needs.

The correct rule is stated per position rather than per trade: after every trade, each side's collateral is brought to the full value of whatever short exposure it now has, with refunds settled before the cash leg so that an account closing a short can pay with the collateral coming back. For two accounts trading from flat this reduces to the obvious rule, which is why the obvious rule looks right.

That is a design error, found by a test, before any customer money existed. It is the argument for the section that follows.

Testing something that must not be wrong

An order book is unusually testable, because determinism means two runs can be compared for equality rather than for plausibility. Clobber uses five kinds of test, and the point is that each one catches something the others structurally cannot.

Invariants, checked after every command: value is conserved per asset per tenant, available balances never go negative, escrow holds exactly the collateral the open interest requires, the book is sorted and never crossed at rest, terminal orders never change again.

Property tests over random command sequences, which check those invariants across shapes nobody wrote by hand.

A second implementation, deliberately slow and obviously correct, compared against the real engine command by command. The rule that makes it worth having: wherever the real engine maintains incremental state, the reference recomputes from scratch. No price levels, one list of orders scanned and sorted. No maintained collateral totals. Holds are a pure function of whichever orders are currently resting, which makes a leaked reservation structurally impossible rather than merely unlikely. It catches the class of bug that incremental state produces, which is the class the real engine is most exposed to.

A differential harness against an established engine, driving the same order flow through both and comparing the book after every command. This is the only check written by people who never read Clobber's specification, which matters: everything else shares its author's misunderstandings. The differences between the two products are catalogued rather than hidden, and the standing rule is that a divergence is a bug in Clobber until it is shown to be a difference in models.

Crash tests: a process killed with SIGKILL under load, repeatedly, on the same log. Not once, because recovering from a recovery is the case an operator actually meets. A cell that crashes tends to crash again, and each restart writes on top of a file the previous life left half finished.

Twelve real bugs have been found by these so far, several of which would have lost or misplaced money. Every one is frozen as a regression case pinned to the seed that found it, because a property test with a fresh seed goes looking somewhere new every run and nothing guarantees it wanders back to a corner that a later refactor reopened.

The lesson that generalises

The most useful thing this project has learned is not about order books.

Both differential harnesses were run against an engine with its time priority deliberately inverted, filling the newest order at a price level instead of the oldest. Both passed.

The comparison logic was fine. The generator was the problem: it drew prices from a range wide enough that two orders from different accounts almost never landed on the same price level, and a book whose levels hold one order each cannot exercise time priority at all. The tests were faithfully checking a book that could not have the bug.

Narrowing the price band fixed it, and both harnesses now fail within a hundred cases against the same injected bug. The injection is now part of the routine: a differential test that cannot fail is not a test, and the only way to know which kind you have is to break the thing on purpose and watch.

The same lesson repeated at smaller scale while building a realistic flow generator. Its first version cancelled orders that had already filled, so 97% of its cancels were refused and the cancel path, which is where book bookkeeping is most likely to be wrong, was barely exercised at all. A generator that mostly produces rejections is testing the rejection path and calling it coverage.

Durability is one number

An acknowledgement means the command is on disk. Not buffered, not queued: fsynced. That is what makes the recovery story simple, because the log is the truth and the last acknowledged command is the last one in it.

Group commit makes this affordable without a timer or a background thread. A writer appends its bytes, then queues behind whoever is currently syncing. That writer flushes everything buffered so far, and everyone behind them finds their own bytes already on disk. The batch is whatever arrived while the previous sync was in flight, which is the correct size by definition and needs no window to be configured.

Measured on a development laptop, whose SSD has an unusually expensive fsync: one writer at a time costs 3.9 ms per command, and thirty two writers in flight cost 0.27 ms each from the same disk. A factor of fourteen, which is the batch doing its job. Pushed to sixty four writers, a second harness gets 6,045 commands a second from the same disk, or 0.17 ms each.

Those points fit a law with an exponent in it: the effective batch grows as a power of the number of writers in flight, so throughput rises as k^a and latency as k^(1-a). Measured here, a is about 0.85 up to thirty two writers and about 0.42 above it, which means the batch stops paying its way exactly where a busy cell would want it to. That exponent, not the matching loop, is what decides which disk a cell needs. The derivation, the measurements and the storage it rules out are in the fundamentals.

Matching itself was for a long time quoted here at 856,000 orders per second, with recovery replaying 2.2 million commands a second. Both figures were measured on a workload in which the book never crossed: every order rested, nothing ever filled, and nothing ever finished. They are the cost of inserting an order, which is a real number under the wrong name. Both were re-measured on a workload that crosses, and the same blind spot turned out to be hiding a defect on the hot path, which is the more useful half of the story.

The shape those numbers describe is the useful part. Matching costs nanoseconds and durability costs milliseconds, so throughput is not a question about the engine, it is a question about how many commands share one fsync. Anyone sizing a system like this should start there rather than with the matching loop.

It is also why Clobber's published documentation promises acknowledgements in single digit milliseconds rather than under one. An earlier draft claimed under a millisecond. That is arithmetically impossible for a durable acknowledgement behind a group commit window, so the documentation was corrected rather than the promise quietly reinterpreted.

Multi tenancy is in the keys, not in a filter

Every key in the engine begins with the tenant. Balances, markets, accounts, orders. A lookup is resolved inside a tenant, so a command carrying an identifier from another tenant does not fail a check, it simply does not resolve.

The distinction matters because a check can be forgotten in one code path. A key structure cannot. Cross tenant access is refused as not found rather than forbidden: telling a caller that an object exists but belongs to somebody else is itself the leak the boundary is there to prevent.

Matching is single homed, and always will be. An order book requires total ordering, so active-active matching across regions is not an engineering trade off, it is a contradiction. Reads scale by relaying the feed; writes go to the tenant's home. The home can move, and moving it is the migration procedure described earlier: halt, snapshot, replay, compare hashes, switch.

The contract is written before the code

The full API is published and versioned before the surface that serves it. That ordering is deliberate and it is the discipline the rest of this document depends on: when the specification is the artifact and the implementation is measured against it, "does this behave correctly" is a question with an answer rather than a matter of opinion. Where the code and the published contract disagree, the contract wins and the code is what changes.

It is also why the engine can be built from the inside out. The ledger came first, then the book, then the matching, then durability, each one gated on the invariants above holding before the next was started. A venue is not a thing you make correct at the end.

Why this is worth renting

The engineering above is not exotic. Every decision in it is available to anyone who reads the same papers and takes the same care. That is precisely the argument: it is a solved problem that stays expensive to solve, and the expense is not the matching loop, it is the year spent discovering that determinism is the answer to failover, that rounding must happen in one place, that a differential test can pass against a broken engine, and that an acknowledgement means fsync.

Clobber exists so that a team building a prediction market can spend that year on the market.

Clobber is a hosted central limit order book: markets over HTTP, matching in microseconds, and every book delta and fill over WebSocket. The full API reference is published at docs.clobberhq.com.