Here is a number: 2,675. For several months it appeared in our own documents as a compression ratio, then as a speedup, then as a planet-scale measurement with a storage figure back-calculated to fit it, then in a slide with an objection pre-empted. Nobody lied. The number was real: a regression reference from a smoke test over one hundred map chunks, produced by an equivalence check that confirmed an optimisation had not changed its output. It travelled without its units and without its extent, and by the fourth document it meant something it had never measured. When fresh measurements contradicted it, it had already been in front of people who mattered.
We did not fix that by being more careful. Care does not scale. We fixed it by making the failure unrepresentable, and this article is about the rule we adopted, because it is the reason you can trust the other numbers on this site and because we think buyers should demand it of every vendor, including us.
The rule
No performance figure is published without three things attached, and the three things are enforced in code, not in a style guide.
- Its quantity kind. A compression ratio and a speedup are different kinds and cannot be compared or converted. A pass rate is not a throughput. The type system refuses to compare across kinds, so a ratio cannot be relabelled as a speedup by accident.
- Its scope. Dataset, extent and configuration, all required, none blank. The extent is what stops "one hundred chunks" from reading as "the planet". Absolute latencies carry their host and date, because they belong to a machine.
- Its harness. Either a runnable test that re-derives the figure, or an explicit marking that the figure was inherited and has not been re-derived. An inherited figure is carried honestly and is not publishable. It stays internal until someone runs it.
The type that carries a figure cannot print a bare number. Its display always emits value, quantity and scope together, so a figure copied out of a log carries its qualifiers with it. Kind-specific implausibility is rejected at construction: a compression ratio below one, a negative speedup, a pass rate outside zero to one. This is the third of a family of constructs in our engine that make an error unrepresentable rather than discouraged: one stops a verdict from being asserted before its test exists, one stops an ordering from being faked after the fact, and this one stops a quantity from drifting.
What it did to our own claims
The rule retired things we had said. A per-record overhead figure with no harness behind it. A latency quoted from a debug build. A "one hundred thousand records verified" line that described an amortisation denominator, not a workload. A compression headline that newer measurements could not reproduce and that its own harness turned out never to have measured as compression. Each retirement is recorded in a grounding document with the reason, and the document is the authority: if a row says do not quote, the number does not appear, however good it looked.
It also changed how the surviving figures read. Our post-quantum stored-data measurements say that a 115-byte record encrypts to 131 bytes whether its key came from a classical key-derivation function or from ML-KEM, and that median per-record times agreed within one percent on both hosts.1 That sentence is longer than a headline. It is also true on any machine you run it on, and the raw logs are linked beside it.
Why buyers should demand this
A number without its quantity is a rhetorical device. A number without its extent is an extrapolation wearing a measurement's clothes. A number without a harness is a memory. Every benchmark dispute we have watched in this industry reduces to one of those three, and every one of them is resolvable before publication by a rule the vendor could have adopted. When you evaluate a data product, ask for the three things. If the answer is a deck, you have learned what the number is worth.
How to hold us to it
Every figure on this site links to its raw output or names its harness. The benchmark and known-answer suites run from source with one command each, and the reproducibility package is available on request so you can produce our numbers on your hardware rather than ours. Where a figure cannot yet meet the rule, we publish the mechanism or the protocol instead of the number.2 Our pre-registered comparison against Oracle's online rekey is the rule applied before a result exists: the method, fixtures and pass criteria are public first.3
What to bring us
- A number of ours you doubt. Tell us which and we will send the harness, the scope and the raw output, or retire it in public if we cannot.
- A number of someone else's you doubt. We will show you the three questions to ask.
- Your workload. Run the reproducibility package on it. If our figures do not hold on your machine, that is a result we want to know before you do.
Write to hello@8braid.com with the subject "Reproducibility package request". You will hear back from an engineer.
Sources and further reading
- 8DB: Post-Quantum Cryptography for Stored Data, Measurements and Method
- 8DB: We Are Measuring Our Migration Against Oracle's Online Rekey
- 8DB: What an Independent Evaluator Should Be Able to Run
- Nosek et al., The preregistration revolution (PNAS, 2018)
Footnotes
-
Scope: release builds on Windows and Linux x86_64, the same cipher on both sides, two hundred samples per arm after warm-up. The full table, hosts, dates and raw logs are on the stored-data evidence page linked below. ↩
-
Figures we would like to publish but have not yet met the rule for include throughput on a large public corpus and head-to-head migration timings. They do not appear on this site until they have a harness. ↩
-
The Oracle online-rekey comparison has not run yet. The method, fixtures and pass criteria are published ahead of any result. ↩
Continue the technical conversation
Where could this help your work?
Bring a research question, a database workload or an application you want to build. Let’s connect the ideas in this article to an evaluation that matters to your team.
Discuss this work