Status: layer 1 implemented; layer 2 implemented for protection declared at
database creation. The storage-encryption primitives, the blob-store
decorator, log-record payload encryption, the keys:<db> key manifest (with
storage format 4), and the process wiring — --storage-key on the transactor,
peer server, and offline commands; corium db create --storage-key;
corium keys status|rotate|rewrap — are implemented. Remaining in layer 1:
backup format 2 (until then corium backup refuses an encrypted database
rather than writing an archive no restore could open) and KMS-backed keyrings
(a key identity resolves through file: or env: today; awskms:,
gcpkms:, and vault: are recognized as key sources and rejected as
unresolvable).
Layer 2 is implemented end to end for the shape of protection Corium's schema
can express today. A class declared in the create-time schema
(:db.protect/ident in EDN, [protect.<name>] in TOML) protects an attribute
from t = 0; the writing peer seals, the transactor validates the cleartext
header and commits with no key, and a reader hydrates exactly the classes its
keyring resolves. What is not implemented, and why:
corium keys protect|unprotect|audit. Corium's schema is create-time and
immutable: attributes are not entities, and there is no alteration path to
hang a forward-only change on. That mechanism is the prerequisite, and the
single largest remaining piece. Everything downstream of it is already
written against a per-attribute protection timeline rather than a single
class, so alterations append to it rather than reshaping anything.:db.protect.scope/entity, and
sealing under it refuses rather than silently binding the wrong subject. It
needs ReserveEntityIds, local upsert pre-resolution, and the
expected_basis_t retry loop described below.--seal-through, SQL predicate rewriting (SQL gets the safe
defaults instead: NULL for an unhydrated value, and no pushdown on a
protected column), and the thin-client v3 contract document. The
per-principal KeyPolicy is implemented on the peer server and pgwire
(ADR-0021): a key-holding
server hands each request only the classes policy grants it.This document specifies two independent layers
— envelope encryption of every durable artifact
(ADR-0017) and per-attribute protection
classes keyed by separate data keys
(ADR-0018) — and fixes the
formats, schema, APIs, and operational surface they need. Nothing here changes
the peer's public interface: a peer still connects, reads storage itself, holds
an immutable Db, and runs queries locally. What changes is that some values
arrive sealed, and a peer hydrates exactly the ones whose class key it
holds.
plaintext datoms
│
┌──────────────────────────┼───────────────────────────┐
│ layer 2: attribute protection (per-class keys) │
│ values on protected attributes are sealed HERE, │
│ in the writing peer, before tx-data leaves it │
└──────────────────────────┼───────────────────────────┘
│ datoms whose v may be Sealed{class, epoch, ct}
transactor pipeline
log · index job · peers
│
┌──────────────────────────┼───────────────────────────┐
│ layer 1: storage encryption (one DEK per database) │
│ log records, index blobs, backups, cached segments │
└──────────────────────────┼───────────────────────────┘
│
disk / object store
Layer 1 protects the medium: a stolen disk, a readable bucket, a copied backup file. Everyone inside the deployment sees plaintext once the bytes are decrypted, which is the property that makes it cheap and invisible.
Layer 2 protects facts from readers: a peer, a peer server, an operator, or the transactor itself may hold the whole database and still be unable to read salaries. It is the layer the "multiple encryption keys" requirement is about, and the only one that survives a compromised process inside the deployment.
They compose without knowing about each other: a sealed value is opaque bytes by the time layer 1 sees it, so layer 1 encrypts and compresses it like anything else, and layer 2 never has to care where the datom is eventually written.
| Adversary | Layer 1 alone | Layer 1 + protected attribute |
|---|---|---|
| Stolen disk, snapshot, or object-store bucket | protected | protected |
| Copied backup archive | protected | protected |
| Operator with storage credentials | protected | protected |
| Compromised peer process without the class key | plaintext | protected |
| Compromised transactor | plaintext | protected (cannot read; cannot forge) |
| Thin client / SQL session without the class key | plaintext | protected |
Query-authorized insider (Allow from the authorizer) | plaintext | protected unless granted the key |
| Network interception | TLS | TLS + sealed |
The insider row holds for a peer by construction — a process is granted a class key or it is not. For a client of a hosted peer server or pgwire server it holds by policy: the surface resolves a per-principal key set per request, so a client reads only the classes the policy grants it, even when the server holds more. That is enforcement by the process holding the plaintext rather than by mathematics; see Peer server, thin clients, SQL.
Explicit non-goals. None of this hides:
Four kinds of key material, in one hierarchy:
| Key | Scope | Lives | Held by |
|---|---|---|---|
| KEK (key-encryption key) | deployment or database | KMS / HSM / operator file; never in Corium | nothing in-process; used to unwrap |
| Storage DEK | one database, per epoch | wrapped, in the keys:<db> root record | transactor, GC, backup, any peer reading storage directly |
| Class key | one protection class, per epoch | resolved by key id from the reader's keyring; never stored in Corium | only processes granted that class |
| Derived subkeys | per blob / per record / per fact | derived, never stored | whoever holds the parent DEK |
The manifest records the KEK per database, not per deployment, so a deployment that later wants per-tenant key isolation — separate KEKs, separately grantable, separately destroyable — already has the field it needs. Deployment-wide is simply the case where every database names the same one.
The asymmetry is the point. Storage DEKs are stored wrapped, because a restore must be able to bootstrap itself from the archive plus KMS access. Class keys are not stored at all: the database records only a key id, and a process gets material by resolving that id through its own keyring. Corium therefore never decides who may hydrate an attribute — the KMS or keyring grant does. That separation is deliberate; see Keys versus authorization.
corium-cryptA new pure-library crate below everything else, next to corium-core in the
dependency order (no tokio in the primitives; the keyring trait is async
because KMS calls are):
/// Opaque, zeroized key material.
pub struct SecretKey(/* [u8; 32], Zeroizing */);
/// A key identity as written in a root record or a class entity.
/// Rendered as a URI: `file:/etc/corium/pii.key`, `env:CORIUM_PII_KEY`,
/// `awskms:arn:aws:kms:…`, `gcpkms:projects/…`, `vault:transit/keys/pii`.
pub struct KeyId(String);
#[async_trait]
pub trait Keyring: Send + Sync {
/// Material for a specific epoch; used to read existing data.
async fn key(&self, id: &KeyId, epoch: u32) -> Result<SecretKey, KeyError>;
/// The epoch new writes use.
async fn current_epoch(&self, id: &KeyId) -> Result<u32, KeyError>;
/// Wrap/unwrap for stored DEKs (KMS-backed rings do this remotely).
async fn wrap(&self, id: &KeyId, epoch: u32, dek: &SecretKey) -> Result<Vec<u8>, KeyError>;
async fn unwrap(&self, id: &KeyId, epoch: u32, wrapped: &[u8]) -> Result<SecretKey, KeyError>;
/// Which key ids this process can resolve at all — the hydration set.
fn key_ids(&self) -> &[KeyId];
}
Shipped implementations, mirroring how TokenVerifier/OIDC already stage a
seam and its concrete backend: StaticKeyring (files, environment variables,
in-memory test keys) in the base crate; KmsKeyring over a small KmsClient
trait behind feature flags (aws-kms, gcp-kms, vault). CompositeKeyring
tries a list in order so one process can take its storage key from a file and
its class keys from KMS.
Primitives:
zeroize on all material, no key in Debug, no key in metrics
or traces, KeyError never quotes bytes. Operators should disable core dumps
on processes holding class keys; the docs say so and the CLI logs a warning
when it detects RLIMIT_CORE unset.One DEK per database per epoch, wrapped under the deployment KEK. Enabled by
--storage-key <key-uri> at database creation; a database created without it
stays unencrypted forever unless migrated by backup/restore.
Today a blob id is the BLAKE3 hash of the plaintext bytes and the caller
computes it before put. Under encryption the id becomes the hash of the
stored (encrypted) object:
object := header ‖ nonce ‖ ciphertext ‖ tag
header := magic "CORIUMB1" ‖ alg:u8 ‖ epoch:u32 ‖ plaintext-len:u64
nonce := BLAKE3_keyed(dek, "corium/blob-nonce" ‖ header ‖ blake3(plaintext))[..12]
AAD := header
ciphertext ‖ tag := AES-256-GCM-SIV(dek, nonce, AAD, plaintext)
id := blake3(object)
Deriving the nonce from the plaintext digest makes encryption deterministic for a given (epoch, content), which is what preserves every property the segment design depends on. The nonce is stored because a reader cannot derive it from the plaintext digest until after decryption; it is non-secret, and any tampering is detected by the AEAD tag.
Truncating the keyed digest to a 96-bit nonce has a birthday collision bound. That is why blobs use GCM-SIV rather than GCM: a collision between two distinct plaintexts does not become the catastrophic key/nonce reuse failure it would be under ordinary GCM. The keyed derivation still makes collisions rare, while GCM-SIV removes collision-freedom as a safety precondition.
The resulting properties are:
put stays idempotent, and re-publishing an unchanged leaf produces the same
id, so structural sharing and incremental publication are untouched.What is lost: cross-database blob deduplication (identical content in two databases with different DEKs is two objects). That is a fair price and, for a multi-tenant deployment, an improvement.
Equality leakage is scoped to one database and storage-key epoch. An observer of ciphertext can recognize a repeated object id in that scope, but cannot test an arbitrary plaintext guess without the DEK or access to an encryption oracle. Different databases and epochs use different DEKs and produce unrelated object ids for the same plaintext.
API shape: encryption is a decorator — EncryptedBlobStore<S: BlobStore> —
so mark_and_sweep, index_blob_children, backup, and every reader see
plaintext and stay unchanged. BlobStore::put(bytes) -> BlobId already makes
the store responsible for computing the id, so the decorator encrypts before
delegating. It also overrides put_if_absent, deriving the ciphertext id before
the presence check rather than using the plaintext digest from the trait's
default implementation. Its contains-then-put race is benign: concurrent
writers produce identical bytes and the same content id.
The decorator holds an immutable snapshot of already-unwrapped storage DEKs.
Keyring and KMS access belong on database open and key-manifest reload, not on
each blob read; a live rotation atomically replaces the process's decorator/key
snapshot. The foundation implementation reconstructs the decorator, and the
process-wiring work must provide that swap without requiring a restart.
Placement in the read stack matters:
peer read → EncryptedBlobStore → SegmentCache → FsStore/S3/Postgres/Turso
The decorator sits above the cache, so the SSD tier holds ciphertext and its digest check is unchanged. This supersedes peer-segment-cache.md's current reliance on host filesystem encryption for cached data.
The frame header (length, checksum flag) and the CRC32C stay cleartext, so range scans, recovery truncation, and the existing framing machinery are untouched. The record payload is encrypted:
payload := header ‖ ciphertext ‖ tag
header := magic "CORIUML1" ‖ alg:u8 ‖ epoch:u32 ‖ t:u64 ‖ nonce:[12]
AAD := "corium/log-v1" ‖ len(lineage):u64 ‖ db-lineage-id
‖ log-version:u64 ‖ header
ciphertext ‖ tag := AES-256-GCM(dek, nonce, AAD, encoded-record)
Binding (log-version, t) in the AAD means a record cannot be replayed at
another basis or moved between the per-lease-version log files that M7 fencing
relies on; binding the lineage means it cannot be moved between databases. The
CRC32C still covers framing (cheap corruption detection during scans); the AEAD
tag is the authenticity check.
t and the key epoch stay cleartext in the header because frame indexing,
recovery truncation, and epoch selection all happen before any key is applied —
and a basis number is metadata the root records already publish. A decoded
record whose t disagrees with its authenticated header is rejected, so the
cleartext copy can never address a record by a number it does not carry.
The nonce is random and stored, not derived from (log-version, t). Unlike
a blob, a log record is not content addressed and nothing requires re-encoding
one to the same bytes, so determinism buys nothing here — while a derived nonce
would repeat whenever a transaction number is re-issued for different tx-data,
which is exactly what happens when an append is torn by a crash before it is
acknowledged and truncated away on recovery. Key/nonce reuse is fatal under
GCM; 12 stored bytes are not. Every binding property above lives in the AAD,
so nothing is given up by making the nonce unpredictable.
What a random nonce does introduce is a budget. Collision becomes a
birthday problem — after q records the probability is about q² / 2¹⁹⁷ — so
a storage epoch must seal fewer than 2³² log records, the standard ceiling
for randomized GCM nonces. That bound is not a counter to maintain: the log
seals exactly one record per transaction, so the records under an epoch are the
span of t it covers. Each manifest entry records the t it opened at, the
next entry (or the current basis) closes the span, and corium keys status
reports the budget from arithmetic on two numbers already present. Rotating the
storage key opens a fresh epoch and resets it; the warning fires at half the
ceiling so the rotation is scheduled rather than urgent. Blobs are unaffected —
their nonce is derived rather than random, which is exactly why they use
GCM-SIV.
Encryption costs about 49 bytes per record — a 21-byte header, the 12-byte nonce inside it, and a 16-byte tag — with framing unchanged.
Root records stay cleartext. They hold no user data — database name, basis
numbers, blob ids, lease state — and RootStore::compare_and_set compares
bytes, which deterministic encryption would only complicate. What a root leaks
is listed under non-goals above.
A new root record per database, keys:<db>, is the key manifest:
KeyManifest {
format-version,
kek: KeyId,
storage-keys: [ { epoch, kek-epoch, alg, state, created-at, opened-at-t,
live-objects, wrapped-dek } ], // active | retiring | retired
classes: [ { class-entity-id, current-epoch, key-id } ], // ids only, never material
}
Each storage key records the KEK epoch its material is wrapped under, not
just the KEK's identity: a KEK rotation retires a KEK epoch while leaving the
data key's own epoch unchanged, so unwrapping needs both numbers.
opened-at-t is the log-record nonce budget above. live-objects is the
per-epoch count the GC mark pass maintains and corium keys status prints; an
epoch retires only at zero. The two are different meters and neither
substitutes for the other: one counts stored objects for GC drain, the other
counts nonces drawn.
Decoding validates more than syntax, because this record bootstraps every
storage access a process makes: epochs must be strictly ascending and ordered
in t, class entries must be unique, exactly one epoch may be active in a
populated manifest, and no content may trail the entries. A count in the record
is a stated length, never an allocation budget — entries are read one at a time
and decoding stops at the first absent line, so a manifest declaring
18446744073709551615 storage keys fails to decode instead of exhausting
memory at open.
DbRoot gains a key-manifest-version field (storage format 4) so a reader
detects an encrypted database before it tries to parse a blob, and fails with
"database is encrypted; no storage key configured" instead of a decode error.
The field is a generation counter, incremented whenever the manifest changes,
so a running process notices a rotation or re-wrap it has not loaded and
refreshes its key snapshot without re-reading the manifest on every
publication. It is owned by corium keys: lease acquisition and index
publication carry the stored value forward untouched, exactly as publication
already does for the live lease fields.
The class table in the manifest is a cache of what the schema already says
(class entities live in :db.part/db, below) so that a process can discover
which key ids it needs before it can read any datoms. It is advisory; the schema
is authoritative.
Backup format version 2:
Content encryption: u32 and Key manifest: bytes.BLOB frames carry stored objects verbatim — no decrypt/re-encrypt — so
backup remains a byte copy and needs no storage key to copy, only to walk
manifests.CKPT transaction records stay log-framed, hence still encrypted.Restoring into a deployment that can unwrap the archive's KEK yields a working database. Restoring where the KEK is unavailable fails cleanly at open. Restoring without the class keys yields a fully functional database whose protected attributes are permanently redacted — which is exactly what you want when shipping production data to a staging environment.
| Rotation | Cost | Trigger |
|---|---|---|
| KEK | re-wrap each DEK in the manifest; no data touched | corium keys rewrap --kek <uri> |
| Storage DEK | new epoch; new writes use it; old objects stay readable under the retained epoch | corium keys rotate --storage |
| Class key | forward-only, see Class key rotation | corium keys rotate --class :protect/pii |
A storage epoch retires only when no live object carries it. The manifest tracks
a per-epoch live-object count maintained by the same mark pass GC already runs,
and corium keys status prints it; a forced full index rebuild is the way to
drain an epoch deliberately.
Two additions, both ordinary data:
;; A protection class: an entity in :db.part/db naming a key, not holding one.
{:db/id "pii"
:db.protect/ident :protect/pii
:db.protect/key "awskms:arn:aws:kms:us-west-2:…:key/2f1c…"
:db.protect/algorithm :db.protect.alg/aes-256-gcm-siv
:db.protect/scope :db.protect.scope/attribute ; or /entity
:db.protect/padding 64 ; optional, bytes
:db.protect/on-missing-key :db.protect.missing/redact ; or /hide, /error
:db.protect/legacy-plaintext :db.protect.legacy/redact} ; or /pass-through
;; An attribute in that class.
{:db/ident :person/ssn
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/protection :protect/pii}
docs/schema-toml.md gains the same field under an attribute's table
(protection = "protect/pii") and a [protect.<name>] section for classes.
Rules the transactor enforces, all without holding any key:
:db/protection may not be combined with :db/index, :db/unique, or
:db/valueType :db.type/ref — this is the "protected datoms cannot be
indexed" rule, made checkable at schema-install time.:db/protection may not be asserted on schema attributes, :db/ident,
:db/txInstant, or any attribute in the reserved id range.:db/protection may be asserted, retracted, or changed on a populated
attribute, and takes effect forward only — see
Changing protection. An attribute that has ever been
protected may never afterwards gain :db/index or :db/unique.:db/cas, may use any form
the attribute has ever had — plaintext, or sealed under any class and epoch
in its protection timeline. A fact can only be retracted by naming the bytes
it was asserted as, and those bytes do not change when the schema does.:db/valueType.Rules 4 and 5 make the schema cache carry a protection timeline per
attribute — the (t, class) pairs its :db/protection datoms already record —
rather than a single current class. The form required at basis t is what the
timeline says at t; the forms accepted for a retraction are every entry in
it.
A new Value variant and a new tag in the sortable encoding (0xA0, above
REF's 0x90; a protected attribute never mixes sealed and plaintext values,
so cross-type ordering is a formality):
pub struct Sealed {
pub class: EntityId, // the class entity — self-describing across fork/restore
pub epoch: u32,
pub vtype: ValueType, // cleartext: needed for redaction rendering and validation
pub body: Arc<[u8]>, // deterministic AEAD ciphertext ‖ 16-byte tag
}
pub enum Value { /* … */ Sealed(Sealed) }
Encoded as 0xA0 ‖ class ‖ epoch ‖ vtype ‖ body, self-delimiting like every
other value encoding — the body uses the same zero-escaped framing as Str
and Bytes — so Datom::key, key_components, and Datom::from_key need no
structural change. The wire codec carries the same fields under tag 0xA6,
because 0xA0 is LIST in its tag space; sending one needs protocol v3.
A keyword value seals its text under the string tag. Reading one back
therefore needs a keyword id the transactor never assigned, so the interner
keeps a reader-local overlay above FIRST_LOCAL_KW_ID: local ids never reach
storage or the wire (keywords travel by name in both), and a keyword the
reader already knows durably keeps its durable id, so a hydrated value and a
query literal still compare equal.
Sealing:
context := class-key-id ‖ epoch ‖ attr-id [‖ entity-id if scope = entity]
AAD := "corium/seal-v1" ‖ context ‖ vtype
body := AES-256-GCM-SIV(key, nonce = 0, aad = AAD, plaintext = encode_value(v))
Type-specific rules:
:db.protect/padding bytes before sealing, with the true length recovered
from the decoded value. It costs storage and removes the length side channel
for short, guessable values.The index key is the datom. A retraction cancels an assertion by sharing the
(e, a, v) byte prefix, and the current-value fold keeps at most one entry per
prefix. Randomized encryption would give the same fact two different byte
representations, and retraction, cardinality-one supersession, cardinality-many
deduplication, and :db/cas would all break — on a transactor that has no key
and therefore cannot compare plaintext.
So sealing is deterministic: identical plaintext in the same context produces identical bytes. Everything the transactor does with values it does bytewise, exactly as today, and it keeps working with no key at all.
The cost is equality leakage, and the class's scope chooses how much:
| Scope | Context | A keyless reader can tell… | Write cost |
|---|---|---|---|
:db.protect.scope/attribute (default) | key, epoch, attribute | …that two entities share a value on that attribute, and can count distinct values (frequency analysis; for a low-cardinality attribute such as a status enum, that is close to plaintext) | none |
:db.protect.scope/entity | key, epoch, attribute, entity | …only that one entity's value repeated over time | an id-reservation round trip and a basis fence |
Entity scope also strengthens integrity: the AAD binds the entity, so a compromised transactor cannot move a ciphertext from one subject to another. Under attribute scope it can. For adversarial-transactor deployments that, more than the equality leak, is the reason to pay for entity scope.
Entity scope and tempids. Sealing needs the entity id, and a tempid does not have one until the transactor resolves it. The writing peer therefore resolves first:
:db.unique/identity attribute are
resolved locally against the peer's own Db (unique attributes are never
protected, so this lookup is exact and local).Catalog.ReserveEntityIds(partition, n)
RPC — a bump of the same monotonic allocator the transactor already owns.
Unused reservations simply leave gaps, which the allocator already tolerates.expected_basis_t (the
protocol-v2 fence, already implemented) set to the basis the local resolution
was computed against. A concurrent write that would have changed the upsert
outcome rejects the transaction and the peer retries.The fence is what makes this safe rather than racy: without it, a concurrent upsert could bind the transaction to a different entity than the one the values were sealed for, and the mismatch would surface much later as a decryption failure.
| Index | Protected datoms |
|---|---|
| EAVT | present — entity access and pull must work |
| AEVT | present — attribute scans must work |
| AVET | absent — :db/index/:db/unique are rejected at schema install |
| VAET | absent — refs cannot be protected |
Consequences the planner and executor must implement:
a is bound" rule still holds.explain says so.index-range and range predicates over a protected attribute are errors,
not silent nonsense — sealed order is byte order, which is not value order.PlannerStats records protected attributes' datom counts but contributes no
distinct-value estimate; the planner treats a bound protected value position
as unbound for selectivity. (Distinct ciphertext counts under attribute
scope would be a real estimate and a real leak; the estimate is dropped
rather than used.)min/max/sum/avg over an unhydrated sealed value raise
QueryError::Protected; count, count-distinct, and grouping work.Buying lookup back: the blind-index recipe. When an application genuinely needs indexed lookup or uniqueness on a protected field, it stores a second, unprotected attribute holding a keyed hash:
{:db/ident :person/email :db/valueType :db.type/string :db/protection :protect/pii}
{:db/ident :person/email-hmac :db/valueType :db.type/bytes :db/unique :db.unique/identity}
The writing peer computes email-hmac = BLAKE3_keyed(blind-index-key, email)
and transacts both. Lookup and uniqueness are then ordinary AVET work, at the
documented price: the blind index is deterministic, indexed, unprotected data,
and it leaks equality by construction. Keeping it a recipe rather than an engine
feature keeps that price visible. (An engine-level :db.protect/blind-index
that derives and maintains the second attribute automatically is a plausible
follow-up; it would not change any of the above.)
Protecting an attribute, unprotecting it, or moving it to another class are all legal alterations, and all of them are forward-only: a datom keeps the form it was asserted in, forever. Nothing rewrites the log, and nothing re-encrypts history, because that is the one thing an immutable database does not do.
That is the consistent answer, and it is also the surprising one — "I protected the attribute" reads as "the data is now protected", and cryptographically it is not. The design's job is to make the gap visible, bounded, and closable rather than to pretend the flip did more than it did. Three mechanisms do that, in increasing order of strength and cost.
| Alteration | Datoms before t | Datoms after t |
|---|---|---|
Protect (assert :db/protection) | plaintext, in the log, the indexes, every peer's memory, and every existing backup | sealed under the new class |
| Unprotect (retract it) | sealed; still need the class key to read, forever | plaintext |
| Re-classify (change the class) | sealed under the old class; readable by holders of the old key | sealed under the new class |
| Rotate the class key | sealed under the old epoch | sealed under the new epoch |
A sealed value carries its class and epoch in its cleartext header, so a mixed attribute needs no schema archaeology to read: each datom says what it needs. That is why re-classification works at all, and why the timeline in schema rule 5 is a validation device rather than a decoding one.
The one hard prohibition is re-indexing. An attribute that has ever held sealed
datoms may never gain :db/index or :db/unique, because AVET would then mix
value-ordered plaintext with byte-ordered ciphertext: range scans would return
silently wrong answers and uniqueness would silently not be enforced. Excluding
the sealed datoms instead would make AVET incomplete, which is worse — a
lookup ref would miss them. Symmetrically, protecting an attribute that is
currently :db/index or :db/unique requires retracting those in the same
transaction; the next publication stops emitting AVET entries for it, and
lookup refs through that attribute stop working, which is usually the part
that breaks application code.
A reader that cannot hydrate an attribute's sealed values does not get its
legacy plaintext either. :db.protect/legacy-plaintext defaults to redact:
a plaintext datom on an attribute that is currently protected is treated
exactly like a sealed value the reader has no key for, following the class's
on-missing-key policy. pass-through restores the literal behaviour for
deployments that want it.
This is policy, not cryptography — the same strength as a ViewFilter, enforced
by the serving process, defeated by anyone who can read segments directly. But
it is immediate, free, and it means the ordinary consequence of protecting an
attribute is that keyless readers stop seeing its values, which is what the
operator expected in the first place. Key holders are unaffected: they see
plaintext for old datoms and hydrated values for new ones, uniformly.
Deliberate wrinkle: this policy is evaluated against the current schema, not
the schema of the time view being read. An as-of before the protection basis
would otherwise hand back the plaintext by time-travelling under the policy.
Schema-as-data says the old view had no protection; confidentiality says the
answer is no. Confidentiality wins, and the exception is documented here
because it is the one place where a read does not see the schema of its own
basis.
Redaction does not help against a reader with storage access, and protection
alone does not seal a value that nobody has re-asserted since. corium keys protect <attr> --class <c> --sweep performs the alteration and then, from a
key-holding peer, walks AEVT and re-asserts every current value in sealed form
(retracting the plaintext datom and asserting its sealed twin, chunked into
bounded transactions, resumable, and idempotent — a value already sealed under
the current class is skipped).
After a sweep the current database value holds no plaintext for that
attribute. History still does, and as-of before the sweep still yields it,
subject to the redaction policy above. A :db/noHistory attribute is the happy
exception: nothing retains the superseded plaintext in any index, so a sweep
leaves it only in the log — which is the closest thing to retroactive protection
that does not involve rebuilding or shredding.
Cardinality-many needs care in both directions: across an unswept transition an
entity can hold both a plaintext and a sealed copy of the same value — they are
distinct datoms with distinct keys, so a key holder sees a duplicate. The sweep
removes it; without the sweep it is a real artifact of a mid-life change and
worth stating plainly. Cardinality-one cannot exhibit this, since supersession
keys on (e, a).
Only two things make a protection change retroactive, and both are heavy:
Shredding is class-granular, so it only expresses "this attribute's history" when the attribute owns its class. The operational rule that follows: give an attribute its own class whenever you might want to re-classify or shred it independently. Classes are cheap; entanglement is not.
The mirror-image hazard is unprotecting: once an attribute has sealed datoms, its class key must be retained for as long as that history matters. Shredding after unprotecting destroys the old values while the attribute reads as unprotected in the schema, which is the most confusing possible state to arrive at by accident.
A change this consequential should not be silent, and its residue should be a number rather than a worry:
[:db/add "datomic.tx" :db.protect/acknowledge-forward-only true], which the
CLI sets for corium keys protect and which a hand-written transaction must
set deliberately. The point is not ceremony; it is that the acknowledgement
is recorded on the transaction entity, so the schema history says who
accepted the semantics and when.corium keys audit <attr> reports the exposure directly: the protection
basis t, how many current values are still plaintext, how many historical
plaintext datoms exist below t, and how many published index roots and
backup archives still contain them. A sweep drives the first to zero; only a
rebuild or a shred drives the rest there.corium keys status prints each attribute's protection timeline, so
"protected since" is answerable without querying the schema history by hand.Sealing happens in the writing peer, in corium-peer's transact path,
before tx-data is encoded onto the wire — the peer already holds the schema, so
it knows every attribute's class:
(a, v) pairs (corium-tx's
expansion is a pure function and already reusable here).Db rather than re-sealing, so
it names the right form even when that form predates the current class (or
predates protection entirely).PeerError::MissingKey(class). Writing is never partial.Because sealing reads the peer's schema and the transactor validates against
its own, a protection change committed between the two rejects the transaction
rather than storing a value in the stale form — the same staleness the
expected_basis_t fence exists for, and the peer retries against the new
schema.
The transactor validates, orders, logs, and indexes sealed values without ever
resolving a key. Specifically it still performs: tempid resolution, cardinality
enforcement, retraction pairing, :db/cas (a bytewise compare of two sealed
values — which works precisely because sealing is deterministic), and every
uniqueness check (which protected attributes never have).
The exception is database functions. A :db/fn running on the transactor
receives sealed values and cannot branch on their plaintext. That is a real
limitation of keeping keys out of the transactor, and the answer is to do that
work in the peer, guarded by expected_basis_t, rather than to hand the
transactor keys. :db/cas on a protected attribute is the one comparison that
does still work.
Values stay sealed inside the engine — in segments, in the live index, in join
keys — and hydration is a property of the read, not of the Db value. That
matters because one peer server serves many principals with different key sets
from the same Db.
corium-peer: ConnectConfig::with_keyring(Arc<dyn Keyring>). A Db
obtained from that connection carries the connection's keyring by default, so
the embedded-peer API is unchanged for an application that just wants its
values.corium-query: ExecOptions gains a hydrator: Option<Arc<Hydrator>>.
Hydration is applied at scan output — a datom leaving an index scan on a
protected attribute is hydrated if the key is available — so predicates,
functions, aggregates, sorting, and pull all see plaintext with no further
changes. A bounded per-connection plaintext cache keyed by ciphertext digest
keeps repeated scans of the same values off the AEAD path.ViewFilter work must:
a result computed with a key must never be served to a caller without it.
Corium's query cache is parse-only — it holds parsed queries, not results —
so there is nothing to key yet. The requirement lands with the first result
cache.When a value cannot be hydrated, the class's :db.protect/on-missing-key
decides, and a request may narrow (never widen) it:
| Policy | Behaviour |
|---|---|
redact (default) | The value binds as Value::Sealed and renders as #corium/redacted {:class :protect/pii :type :db.type/string} in EDN, NULL in SQL, the sealed tag on the wire. Structure is visible; the value is not. |
hide | The datom is filtered out of scan results entirely, so [?e :person/ssn ?s] binds nothing and the entity drops out of that join — the same shape as an attribute denylist ViewFilter. |
error | The read fails with QueryError::Protected(class). For deployments that would rather see a loud failure than a quiet hole. |
The same three policies cover legacy plaintext — a datom asserted before the
attribute was protected, when :db.protect/legacy-plaintext is redact (see
Changing protection). Both checks happen at the same
place in the scan, against the same key set: the question "can this reader have
this attribute's values?" is asked once, and its answer does not depend on which
side of a schema change the datom fell on.
Under every policy, an unhydratable value never satisfies a value-position constant and never satisfies a predicate. It binds, or it disappears, or it raises — it never matches by accident.
Peer server holding keys. Hydrates per request. The key set for a request
comes from a KeyPolicy: Principal → [KeyId], which is where this meets
authorization: the ReBAC policy names key ids the same way it names view
filters. Plaintext then travels to the thin client over TLS.
A view carries the key ids it grants on :authz.view/key, and the filter
type is optional, so a view may restrict attributes, keys, or both:
;; A key-only view: restricts no attribute, grants one class key.
{:authz.view/name "pii-reader" :authz.view/key ["kms:corium/pii"]}
{:authz.binding/relation "hr" :authz.binding/object "database:people"
:authz.binding/view "pii-reader"}
Key grants combine across successful paths the same conservative way view filters do — by intersection — so holding one more relation never widens a key set. The serving process then hands the request the subset of its own keyring those ids resolve to: policy can never grant a key the process does not hold.
What a decision that names no key means is the surface's key policy mode:
| Mode | A decision naming no key id | Default when |
|---|---|---|
strict | grants no class key | a Guard is configured |
server-wide | grants the process's whole keyring | authorization is disabled |
Deriving the default from the guard is what keeps a single-tenant or
embedded deployment working exactly as before while making the multi-tenant
one safe by default. Note that :authz.binding/unfiltered grants full
attribute visibility and no keys: keys are named by key id, and that
binding names none — a relation that must read protected values names them.
This is authorization, not cryptography. A peer server under
strictstill holds the plaintext; it is choosing not to disclose it. The cryptographic floor is a process that was never given the key, which is still the right configuration for a genuinely less-trusted peer server: it answers every query that does not touch a protected value identically, and protected values come back redacted, hidden, or refused per class policy. An embedded peer (LocalPeer, thecorium-clientfluent API) is not affected either way: its keyring belongs to the one application process holding it.
Peer server in seal-through mode (--seal-through). Returns sealed values
and lets the thin client hydrate with its own keyring — end-to-end protection
for languages using the thin protocol, at the cost of client-side key
distribution. This requires the sealed tag in the wire codec and a
thin-client protocol version bump (v3); a v2 client that would receive a
sealed value gets FAILED_PRECONDITION rather than bytes it cannot name.
SQL / pgwire. A protected column keeps its declared type and reports
NULL when unhydrated. The planner refuses pushdown of any predicate over a
protected column — otherwise a WHERE clause would answer the question the
NULL projection refuses — and will rewrite = to a sealed-bytes comparison
when the session holds the key and the class is attribute-scoped (not yet
implemented). corium sql prints <redacted>. Session key sets come from the
same KeyPolicy as the peer server, resolved per statement from the SQL
session's own principal: PostgreSQL has no bearer-token field, so the
password carries the caller's token (see
ADR-0021).
Rotation is forward-only. New assertions seal under the new epoch; existing
datoms keep the old one, because the database is immutable and history is not
rewritten. A reader needs every epoch it wants to read, and
corium keys status --class :protect/pii prints the epochs in use with datom
counts per epoch.
An epoch pins the whole crypto parameter set — key material, algorithm,
scope, and padding — because all four are baked into existing ciphertext and its
AAD. Changing any of them on a class therefore mints a new epoch and is
forward-only in exactly the way a key rotation is; a class whose scope changes
from attribute to entity protects new values more strongly and cannot
retroactively protect old ones. The class's read policies —
:db.protect/on-missing-key and :db.protect/legacy-plaintext — are not crypto
parameters, so they change freely and apply immediately to every epoch.
The corollary is a genuinely useful primitive: destroying a class key makes
its ciphertext unrecoverable, including in history, in every backup, and on
every peer. For an immutable database that is the practical answer to
"delete this data" — excision by key destruction rather than by rewriting the
past. It is class-granular, which is coarse: shredding :protect/pii shreds it
for everyone.
Per-subject erasure needs per-subject keys. The shape is a
:db.protect.scope/entity class whose :db.protect/key names a key
namespace, with the keyring resolving <namespace>/<entity-id> to material
held in a key table (a small Corium database, or Vault) where deleting one row
destroys one subject's key. That is a real design, with real costs — a key
lookup per subject on the read path, and a key store that is now a durability
dependency — and it is deliberately out of scope for v1, sketched here so
the class model does not have to change to accommodate it later.
Corium now has two ways to keep a reader from seeing a fact, and they are not redundant:
ViewFilter (authz) | Protection class (keys) | |
|---|---|---|
| Enforced by | the serving process | mathematics |
| Survives a compromised peer? | no | yes |
| Survives a stolen backup? | no | yes |
| Granularity | attribute, entity, predicate | attribute (class) |
| Changed by | a policy transaction, effective immediately | key distribution, and only forward |
| Cost | a filter on the read path | sealing, scan-only access, no indexing |
Use policy for "who should" and keys for "who can". The natural deployment
combines them: the ReBAC policy denies the query outright, and the key was
never distributed, so a policy bug does not become a disclosure. KeyPolicy
lets one policy database express both, but they remain independently enforced —
a peer that ignores the policy still cannot decrypt.
This also gives the deferred AllowFiltered work a cheaper first customer:
attribute-level redaction driven by an absent key is enforceable in the scan
today, with no executor predicate.
# Storage encryption, at database creation.
corium db create people --schema schema.toml --storage-key awskms:arn:…:key/2f1c…
# A protection class and an attribute in it (ordinary schema, ordinary transaction).
corium keys class add :protect/pii --key awskms:arn:…:key/9ab3… --scope entity
# Protect an existing attribute: alter the schema, then seal the current values.
corium keys protect :person/ssn --class :protect/pii --sweep
corium keys audit :person/ssn # plaintext still current / in history / in backups
corium keys unprotect :person/legacy-note # forward-only; old values stay sealed
# Who can read what.
corium keys status # timelines, epochs, live-object counts, key ids
corium keys rotate --class :protect/pii # forward-only; new writes use the new epoch
corium keys rewrap --kek awskms:arn:… # KEK rotation; no data rewritten
corium keys shred :protect/pii # destroys material; irreversible; audited
# Processes declare which keys they hold.
corium transactor --data-dir … --storage-key file:/etc/corium/storage.key
corium peer-server --db people --storage-key file:… --key :protect/pii=awskms:…
corium peer-server --db people --storage-key file:… --seal-through
Environment overrides CORIUM_STORAGE_KEY, CORIUM_KEYRING, and
CORIUM_KEYS follow the existing CORIUM_* conventions.
The long-running ones — the sweep, an epoch drain, a rewrap across a large
database, and the audit that must precede a shred — are hour-scale jobs that
need a process outliving the shell that started them, and the sweep additionally
needs to hold a class key while it runs. Those are duties of the proposed
operator peer service: corium keys protect --sweep
submits a job and follows it, key custody for the job is an explicit opt-in
grant recorded with it, and shred requires a fresh plan plus a second
approver. Without a service configured the commands still run in-process, which
is fine for a small database and is exactly what you do not want for a migration
measured in hours.
Failure modes, all of which must be distinguishable in logs and metrics:
| Condition | Behaviour |
|---|---|
| Encrypted database, no storage key | refuse to open, name the manifest's key id |
| Storage key resolves but epoch missing | refuse to open; that epoch's data is unreadable |
| Class key missing | reads follow on-missing-key; writes to that class refuse |
| Class key wrong (unwrap succeeds, AEAD fails) | QueryError::Protected with the class and epoch, never a decode error |
| KMS unreachable | cached material keeps serving; new epochs fail; corium_keys_unavailable gauge set |
| Manifest re-wrapped under a KEK this process cannot resolve | cached material keeps serving (a re-wrap does not change the data keys); corium_keys_unavailable gauge set, reported by corium keys status |
| Manifest opened an epoch this process cannot load | writes refuse, naming both epochs; reads, publication, and the lease continue. Sealing on would draw nonces under an epoch whose budget has stopped counting them |
| Assertion in the wrong form for the attribute's current state | transaction rejected at validation |
| Retraction naming a form the attribute never had | transaction rejected at validation |
| Protection altered without the acknowledgement | transaction rejected, with the remedy in the message |
| Legacy plaintext read by a keyless reader | follows legacy-plaintext (default redact); counted in corium_legacy_plaintext_reads_total |
Metrics: corium_seal_ops_total{op,class}, corium_seal_errors_total{kind},
corium_hydrate_cache_{hits,misses}_total, corium_blob_decrypt_seconds,
corium_key_epoch{key_id}, corium_keys_unavailable. Key ids appear as labels;
key material never does, and the audit sink already used by corium-authz
records key grants, rotations, and shreds.
DbRoot.key-manifest-version), backup format 2,
thin-client protocol v3 (sealed value tag). Each is additive and each is
version-checked the way the existing formats are; an older reader meeting a
newer artifact gets the existing upgrade error, not a parse failure.--storage-key.:db/unique must be retracted at the same time.Value::Sealed is a new variant on an enum matched exhaustively in roughly
thirty places across corium-query, corium-sql, corium-pgwire,
corium-cljrs, corium-ffi, and corium-cli. That mechanical blast radius
is the single largest implementation cost of layer 2, and it is deliberate:
making every consumer confront the variant is how "this value might be
unreadable" stops being forgettable.Steps 1–4 are done; 5–8 remain, in that order. Step 5 gates the rest of the protection story, because everything it covers presumes a schema that can be altered.
corium-crypt. Primitives, KeyId/SecretKey/Keyring,
StaticKeyring, deterministic sealing, derivation, zeroization. Pure
library, property-tested in isolation.EncryptedBlobStore decorator and the id change; log record
payload encryption; keys:<db> manifest and storage format 4; cache
placement; backup format 2; --storage-key on every process; corium keys init|status|rotate|rewrap. Deliverable acceptance: a byte scan of a
populated data directory and blob store finds no sentinel plaintext, and a
full backup/restore round-trips across a DEK rotation.Value::Sealed, the 0xA0 encoding, class entities, the
per-attribute protection timeline in the schema cache, schema validation
rules 1–6, peer-side sealing, transactor keyless validation, EAVT/AEVT-only
membership, planner rules.ExecOptions, redaction policies
(including legacy plaintext), query cache keying, pull/entity/datoms
surfaces, corium-cljrs rendering, console output.:db/index/:db/unique as a supported
alteration (the prerequisite for protecting an indexed attribute), the
forward-only transitions and their acknowledgement, corium keys protect|unprotect|audit, and the resumable, chunked sweep.ReserveEntityIds, local upsert pre-resolution, the
expected_basis_t fence, and its retry loop.KeyPolicy (done), --seal-through,
thin-client protocol v3, SQL/pgwire redaction and pushdown rules,
schema-TOML fields.Acceptance tests, beyond unit coverage:
key_components prefix.as-of before the sweep still does; and re-indexing an
ever-protected attribute is rejected.:db/fn and protected values. Keeping keys out of the transactor means
database functions cannot branch on protected plaintext. An opt-in "transactor
holds this class's key" mode would restore it and give up the strongest
property in the threat model. Not proposed; recorded because deployments will
ask.as-of. Evaluating the policy against
the current schema rather than the view's is the safe choice and the one
inconsistency with schema-as-data in the whole design. A future :db/protection
that participates in time views properly would need reads to consult two
schemas and explain which one answered.:db.protect/lineage marker — and the answer sharpens under multi-tenancy,
where a shared class id is the difference between "one key per tenant" and
"one tenant's peer can open another's values if it ever obtains the datoms".
One property already helps: the seal context binds the attribute's entity
id, which is per database, so two databases sharing a class key still
produce different ciphertext for the same plaintext and no cross-database
equality oracle exists. Fork and restore preserve ids, which is exactly where
determinism must hold.Can you improve this documentation? These fine people already did:
Casey Marshall & ClaudeEdit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |