Every serious effort to make AI trustworthy has produced a benchmark. Models are scored on accuracy, robustness, safety and bias. Vector stores are scored on recall and queries per second. Retrieval pipelines are scored on whether the right passage came back. Between those two layers sits the question that decides whether an agent's answer can be acted on, and nobody scores it: does the store the agent reasons over know where its facts came from, how strongly each is supported, whether two sources disagree, and the difference between a fact that is absent and one that is unknown?
Call it epistemic integrity, and notice that it is a property of the data layer rather than of the model. A perfectly calibrated model reading from a store that flattens confidence and hides provenance will be confidently wrong at exactly the rate the store misleads it. The industry needs a benchmark for that layer, it has to be neutral to be worth anything, and we are proposing one. This article is the outline and the invitation.
What the benchmark would measure
Four families of metric, each scorable on any database, vector store or retrieval stack, with fixtures anyone can run.
- Provenance completeness. For a set of retrieved facts, what fraction can be traced to a source record, with the derivation steps between, and can that trace be verified by a party who does not operate the store? Scored from zero (a passage with no origin) to full (content-addressed source, derivation chain, independently checkable integrity proof).
- Confidence calibration. When the store attaches a confidence to a fact, does the confidence predict correctness on a held-out truth set? A store that reports no confidence scores as if every fact were certain, which is what it is telling the model.
- Typed-absence handling. Given queries whose true answer is missing, redacted, stale, or outside the querier's authorisation, does the store return the right kind of nothing, or does it return null and let the model guess? Scored on how many distinct absence states the store can express and how often it uses them correctly.
- Retrieval honesty. Under conflicting sources, does the store surface the conflict or silently resolve it? Under weak evidence, does it lower confidence or drop the record? Under an out-of-scope question, does it abstain? Scored against fixtures where the honest behaviour is known.
None of these measures speed. Throughput benchmarks exist and are good at what they do. This one measures whether the answer deserves to be believed.
Why it has to be neutral
A benchmark a vendor cannot lose is marketing. The value of this one depends on three design rules we are committing to before any score exists. The metrics must be definable in terms any store can implement, so a relational database, a vector index and a graph store can each be scored, and each can improve. The fixtures and scoring code must be public, so a result can be reproduced by the party who most wants it to be wrong. And the first published scores must include systems that are not ours, chosen because they are what the industry actually runs.
We are aware of the obvious objection: 8DB carries confidence, provenance and typed absence as storage columns, so we would expect to score well on a benchmark that measures them. That is why the design has to be public before the scores, and why we want competitors and sceptics designing the fixtures. If the metrics are right, a store that scores well on them is a store an agent should be built on, whoever makes it. We expect to be scored first, and to learn things we would rather not.1 That is the point.
Where it would live
The right home for a benchmark like this is a community body, not a company. We are preparing a proposal to the working groups that already maintain AI benchmarks.2 The metric definitions, fixture designs and scoring methodology will be published under an open licence as they stabilise, and this page will link to them.
What to bring us
- Your evaluation team. If you evaluate retrieval or agent systems and have wanted a data-layer metric, help define these four before they are fixed.
- Your store. If you build a database, vector store or retrieval stack and want to be scored on a level field, tell us what fixtures would be fair to you.
- Your failure cases. The confidently wrong answers your agents have produced are the fixtures this benchmark needs.
Write to hello@8braid.com with the subject "Epistemic-integrity benchmark: participate". You will hear back from an engineer.
Sources and further reading
- MLCommons and the AILuminate benchmark
- NIST AI Risk Management Framework
- 8DB: The Database Is Part of the Model
- 8DB: Every Number on This Site Has a Harness
Footnotes
-
The confidence, provenance and typed-absence mechanisms in 8DB are exercised in our independent evaluation package as supporting demonstrations, not formal acceptance claims, and their calibration against held-out truth has not been measured in a form we would publish. A benchmark like this would measure it. ↩
-
We are surveying the existing suites to state precisely where the gap is rather than assert it. If the gap turns out to be smaller than we think, the proposal shrinks to fit. ↩
Continue the technical conversation
Where could this help your work?
Bring a research question, a database workload or an application you want to build. Let’s connect the ideas in this article to an evaluation that matters to your team.
Discuss this work