An evaluator's job is to find out whether a product does what it says, under conditions the evaluator controls, without having to trust the vendor. Most products make that job harder than it needs to be. Claims are broad, evidence is a slide, the test environment is the vendor's, and the words "validated" and "compliant" appear in places where no validation or compliance has happened. The evaluator ends up assessing a narrative.
We built the evaluation package for 8DB the other way round, and this article describes its shape. It is written for the people who do this work: independent evaluators, program security staff, assessors and authorising officials, and the integrators who have to carry a product's evidence into a system package. The shape is not specific to us. It is what a product should look like when it expects to be checked.
Start from what the evaluator receives
The evaluator receives one pinned artifact. It contains a stripped runner, frozen baselines and public known-answer vectors. It does not contain source code, build manifests or repository history, so nothing in the evaluation depends on reading or trusting our code. The artifact carries a manifest hash; the evaluator verifies it before running anything, records the harness source identity and runner build hash, and does not rebuild or substitute a binary. If the version and catalogue listing do not match the pinned catalogue, the evaluation has not started.
The run happens on a host the evaluator controls, network-isolated, with commands, timestamps, operating system and CPU, standard output and error, exit codes, result JSON and artifact hashes preserved as the run record. The workbook is the authoritative record; raw machine output is attached without editing.
Claims small enough to test
The package submits four formal claims, not forty. Each has one bounded statement, named harnesses, controlled preconditions, objective acceptance criteria and explicit non-claims. Anything that does not meet that bar is not a formal claim, however true it may be.
| Claim | Bounded statement | Acceptance |
|---|---|---|
| Unified, safe-by-default sealed storage | In one tested store and write path, multiple data shapes persist and return byte-correct results, persisted content is sealed so known plaintext markers are absent from raw storage, and committed data recovers after a simulated crash | Every listed harness passes and matches its frozen baseline |
| Offline cryptographic governance | Label and clearance boundaries and policy or epoch changes are enforced through key derivation and decryption outcomes, with core operations executable while the evaluator network is disabled | Every listed harness passes; unauthorised or stale-policy access is rejected in the tested cases; the run is performed under approved isolation |
| Post-quantum-aligned authenticated sealed-transfer primitive | The tested primitive uses ML-KEM-1024 and ML-DSA-87 operations and rejects the included tamper, wrong-key, man-in-the-middle and replay cases; the payload can carry already-sealed ciphertext without exposing application plaintext | Every listed harness passes and matches baseline; acceptance is limited to the primitive |
| Dataset-level integrity with compact proofs | The engine authenticates a dataset with one ML-DSA-87 signature over a SHA-384 Merkle root, validates a logarithmic inclusion proof, and rejects tampering through an independent raw verifier path | Both harnesses pass and match baseline, including independent verification |
Read the acceptance column again. There is no threshold that depends on the host, no performance number, no competitive comparison. A throughput harness exists in the catalogue, and it is deliberately excluded from acceptance because its output varies with hardware.1
Non-claims are part of the claim
Every formal claim carries the list of what it does not assert.2 Non-claims are not modesty. They are the part of the package that tells an evaluator where to stop, and they are what keeps a product PASS from being misread as a system authorisation.
Dispositions that mean something
Four dispositions, defined in advance. PASS: every required harness passes, matches the frozen baseline, no required subtest is skipped, artifact identity matches, no uncontrolled deviation. FAIL: a required harness executes under controlled conditions and fails or mismatches for a reason attributable to the product. INCONCLUSIVE: an artifact, baseline, vector, environment, identity or procedure is missing or changed, or a required subtest skipped. NOT TESTED: execution did not begin or the claim was deferred.
The claim set passes only if all four claims pass. Supporting demonstrations never compensate for a failed or inconclusive formal claim. A missing vector is INCONCLUSIVE, not PASS. These rules remove the room in which optimistic readings grow.
Evidence has classes
Not all evidence is equal, and the package says which is which.
- Formal. Directly required for one of the four claims: named harnesses and frozen baselines. Only this class can produce PASS, FAIL or INCONCLUSIVE.
- Corroborating. Supports technical confidence but is not the acceptance mechanism. Our NIST ACVP demonstration-server dispositions for ML-KEM-1024 and ML-DSA-87 sit here.3 Corroborating evidence explains confidence; it cannot cure a failed formal test.
- Demonstration. Useful capability outside the current approval surface: throughput, deduplication, confidence and provenance features. Recorded separately, no effect on disposition.
- Planned. Anything that needs a new artifact, environment or authority: a target-hardware build, a production transport, module validation, additional modalities, competitor benchmarks. Must never be described as validated.
When a vendor's marketing puts a class-four item in class-one language, the evaluator has learned something about the vendor.
Accurate security language, pre-agreed
The package includes a table of what may be said and what must not be implied, so the evaluator and the vendor are reading the same words.4 At rest: AES-256-GCM-SIV write-time sealing is exercised and raw media is scanned for known plaintext markers. Post-quantum: ML-KEM-1024 and ML-DSA-87 parameter sets match FIPS 203 and FIPS 204 and are exercised. Key handling: per-page data keys are derived on demand and not persisted in exportable form.
We hold ourselves to that table in public as well. The words "validated" and "certified" do not appear on our site attached to results from a demonstration server.
What a pass produces
A successful evaluation yields an immutable evidence bundle and a bounded claim disposition. Those feed an OSCAL component definition, control implementation narratives and downstream authorisation packages. The authority boundary is preserved: product evidence reduces duplicate work, and the authorising official still evaluates the configured system, mission, environment and residual risk. Change control follows the artifact digest. A new digest means an impact assessment and a rerun of the catalogue unless a narrower regression is approved, and a passing product evaluation does not extend to a differently configured deployment, wrapper service, protocol or hardware platform.
Why we publish the shape
Programs facing the post-quantum transition, disconnected operations and multi-source data fusion are about to evaluate a great many products. Most of the evaluation cost is in arguing about what was claimed. A package with bounded claims, named harnesses, frozen baselines, four dispositions and explicit non-claims moves that argument to before the run, where it is cheap. We would rather be measured against a hard, clear standard than described against a soft one, and we think evaluators should ask every vendor for the same.
What to bring us
- Your evaluation criteria. If your programme has its own disposition rules, evidence classes or control mappings, send them and we will map the four claims and their harnesses onto them.
- A request for the evaluator bundle. The pinned, source-free artifact and its baselines are available to evaluators and to the integrators supporting them.
- The claim you think is missing. If a capability you need is in our demonstration or planned class, tell us; that is how items move into the formal class with harnesses attached.
Write to hello@8braid.com with the subject "Evaluator bundle request". You will hear back from an engineer.
Sources and further reading
- NIST FIPS 140-3: Security Requirements for Cryptographic Modules
- NIST SP 800-53 Rev. 5: Security and Privacy Controls and SP 800-53A Rev. 5: Assessing Controls
- NIST OSCAL Component Definition Model
- NIST FIPS 203: ML-KEM and FIPS 204: ML-DSA
- RFC 8452: AES-GCM-SIV
- 8DB: Post-Quantum Cryptography for Stored Data, Measurements and Method
Footnotes
-
Performance is recorded as a demonstration, not a gate. No throughput figure contributes to any disposition. ↩
-
The non-claims, in full. The sealed-storage claim does not assert that every modality is complete or optimal, or that plaintext can never appear anywhere; it asserts what the harness tests. The governance claim does not assert a complete mandatory-access-control certification, or that a root secret, entropy source or custody process is unnecessary. The transfer claim does not validate any production transport, network, mesh or radio, and the package says so in the same paragraph that states the claim. The integrity claim does not turn a measured overhead figure into an approval threshold until the benchmark is reproducibly packaged. ↩
-
The package states in the same sentence that the NIST ACVP demonstration server does not issue a CAVP certificate. These dispositions are corroborating evidence only. ↩
-
What must not be implied, from the same table. At rest: the write-time sealing is not full-device sanitisation, not proof against every possible leak, and not a validated module. Post-quantum: exercising ML-KEM-1024 and ML-DSA-87 is not a CAVP certificate, not FIPS 140-3 module validation, and not complete CNSA 2.0 compliance. Key handling: on-demand derivation is not "no keys exist" and not "no trust anchor is needed". ↩
Continue the technical conversation
Where could this help your work?
Bring a research question, a database workload or an application you want to build. Let’s connect the ideas in this article to an evaluation that matters to your team.
Discuss this work