Corium needs a declarative schema update path that compares a desired schema file with the schema installed in a database, explains the operational impact, and applies only changes whose preconditions still hold. The first user-facing surface is:
# Inspect only. This is the default.
corium schema update people --schema schema.toml
# Apply the exact plan after reviewing it.
corium schema update people --schema schema.toml \
--apply --plan <plan-digest>
schema update is deliberately plan-first. Reading a schema file is not
permission to remove data, collapse cardinality, or reinterpret values. A plan
is deterministic at an observed database basis and carries a digest. A schema
change or failed safety precondition invalidates it. Ordinary data writes do not
automatically make an additive plan stale.
This design covers attribute schema. TOML [[entity]] blocks are authoring
groups that supply keyword namespaces. Corium does not persist entity types.
Adding an empty group therefore changes no database state, while adding a group
with attributes is a set of attribute additions.
The decision is recorded in ADR-0020.
Schema is data, and a schema change is a transaction. The pieces:
corium_db::bootstrap installs the schema vocabulary — :db/ident,
:db/valueType, :db/cardinality, :db/unique, :db/index,
:db/isComponent, :db/noHistory, :db/doc, :db/protection, and the
:db/retired marker — in every database, at Datomic's ids where Datomic has
the same attribute. An attribute's metadata is stored under these on the
attribute entity.corium_db::schemadatoms::derive folds those datoms back into a Schema and
Idents. Db::with_transaction_at runs it over the whole record before it
indexes any user datom, so a transaction that installs an attribute and uses
it is legal by construction rather than by datom order.Db, every holder reaches the same schema
from the same log: the transactor deriving a commit, a peer applying a tx
report, and recovery replaying the log all converge without a side channel.Db::apply_transaction drops the incremental index fold when the schema
generation advances. Extending the parent's fold under the parent's schema
would leave an attribute that just gained AVET covered for post-change datoms
only — a silently partial index that reads as authoritative.corium_forms::schemaform::schema_from_edn still assigns ids positionally at
CreateDatabase, so embedded, WASM, authz-bootstrap, and fixture databases
stay reproducible. That schema is the pre-basis seed; updates allocate
above the durable maximum instead.:db/ident does double duty, and the two uses are told apart by partition. On
a :db.part/db entity it is schema — the name an attribute is known by — and
only a schema update may write it. On any other entity it is an ordinary name,
which is how a database function or an enumerated value is labelled, and
ordinary transactions keep writing it. Every other vocabulary attribute is
refused to Transact outright, as is the schema-update audit trail: a
transaction must not be able to claim it was a schema update.
Because the vocabulary is engine-owned, a schema file that declares one of
these attributes is rejected with a distinct EngineAttribute error rather
than a confusing duplicate-ident error. Redeclaring it could not change it and
could only disagree with it.
Both TOML and EDN inputs normalize to an ordered map keyed by canonical ident. The normalized form contains value type, cardinality, uniqueness, index, component, no-history, documentation, and (when attribute protection is enabled) protection-class properties. Installed state also carries facts the desired file cannot erase, such as whether an attribute has ever been protected. Attribute entity ids are taken from the installed schema and never inferred from file order after initial creation.
Matching is exact by ident. The planner must not guess that one removed ident and one added ident are a rename, even when their definitions match. An incorrect rename aliases two meanings permanently. A future explicit migration directive can rename an ident while preserving its attribute entity id. Until then, the safe recipe is add, copy, cut over, and retire.
By default, a desired file manages the declarations it contains. Installed
user attributes absent from the file are reported as unmanaged. The command
does not change them. --prune changes absent attributes into retirement
requests. Engine attributes are never managed by a user file. This makes it
safe to update a database from a partial schema while still supporting complete
manifests.
schema-version remains the TOML file-format version. It is not a database
migration number. A later optional schema-revision may provide an
application-owned monotone label, but correctness depends on the installed
schema and plan digest, not that label.
The planner produces property-level changes rather than an attribute-level
changed flag. A single attribute can, for example, need both uniqueness
validation and AVET backfill. Every change reports:
Impact analysis is server-side or peer-local against a fixed Db. The initial
implementation can scan AEVT for one affected attribute. Later index and
statistics support can make the same questions cheaper without changing plan
semantics. Counts are exact unless explicitly labeled estimates. Samples are
bounded. The counts are not bounded.
The plan uses four execution classes:
| Class | Meaning | Typical examples |
|---|---|---|
additive | No existing fact must be inspected or rewritten | add an attribute, or change cardinality from one to many |
validate-reindex | Existing facts remain valid, but a bounded scan, constraint validation, or covering-index rebuild is required | add index, add unique with no duplicates, or change uniqueness mode |
rewrite | Current facts must change before the desired schema can become active | resolve many-to-one conflicts, copy values to a replacement typed attribute, or retract current values during retirement |
destructive | Information or historical interpretation would be lost | hard deletion, excision, or pretending old values always had a new type |
Risk is reported separately from execution class. Changing isComponent can
be metadata-only yet high impact when many live refs will acquire cascade
semantics. Conversely, an AVET backfill may be operationally expensive but
semantically low risk.
| Change | Inspection | Planned behavior |
|---|---|---|
| Add attribute | ident collision, protection consistency | Allocate a stable db-partition id above the durable maximum and install it atomically. No entity-group record is created. A declaration combining protection with index, unique, or ref is blocked here, not at commit. |
| Remove attribute from file | none by default. Live/history counts with --prune | Report as unmanaged. With --prune, retire it as a high-risk validate-reindex change. Never hard-delete it. Optional current-fact cleanup is a separate rewrite step. |
cardinality one -> many | current count for reporting | Online additive change. Existing values already satisfy it. |
cardinality many -> one | entities having more than one current value | If none conflict, validate and apply. Otherwise require a rewrite with an explicit value-selection policy. Never choose a winner implicitly. |
Add index | current datom count, index capacity estimate, ever-protected flag | Reject an attribute that has ever been protected. Otherwise activate only after AVET backfill is complete, or keep the planner on AEVT until completion. |
Remove index | current AVET coverage | Stop planning new reads through AVET. A rebuild reclaims stale coverage. The planner never treats stale coverage as authoritative. |
Add unique | duplicate value groups, their entities, and ever-protected flag | Reject an attribute that has ever been protected or has duplicate values. Otherwise install a pending constraint, backfill AVET, and activate after tail validation. |
Remove unique | whether index remains requested | Change future write semantics. Rebuild AVET only when coverage is no longer requested. |
unique identity <-> value | no data rewrite. Report all users | Change upsert/conflict behavior only after explicit acknowledgement. |
Toggle isComponent | count live refs and component fan-out/cycles | Change future pull and retract-entity semantics. Existing facts are not rewritten. The high-impact semantic change requires acknowledgement. |
Toggle noHistory | presence of existing history and omitted-history interval | Forward-only. Enabling does not erase old history. Disabling cannot reconstruct history already omitted. The plan states the incomplete interval. |
Change doc | none | Add, replace, or retract documentation as a metadata-only change. |
| Protect, unprotect, or re-classify | current index/unique/ref state, live plaintext/sealed counts, protection timeline, lookup-ref use | Forward-only high-risk validate-reindex change requiring protection-forward-only. Protection requires removing index/unique in the same schema transaction and is forbidden for refs. The plan reports lookup-ref breakage and offers the current-value sweep as separate rewrite work. |
| Change value type | current value counts and types | Direct mutation is destructive and never executable. The planner emits a separate replacement-attribute recipe whose copy and cleanup steps are rewrite work under an explicit conversion. |
| Rename ident | exact explicit source/target directive | Preserve the attribute id and history. Never infer from similarity. Deferred until the directive is specified. |
| Hard delete/excise | live and historical datom count, backup reachability | Unsupported by schema update. Excision is a separate destructive facility with its own design and approval. |
index and unique activation use a two-stage state: requested, then ready at
a basis. The query planner and uniqueness validator must not assume coverage
before the backfill reaches that basis. A small database may finish both stages
inside one command. This distinction prevents that optimization from becoming
a correctness assumption.
Uniqueness has an additional pending state that closes the scan/activation
race. The transactor validates current values and installs the pending
constraint in one turn of the writer queue at scan_basis. New writes then
enforce uniqueness through a correct AEVT fallback while AVET is rebuilt. The
activation turn re-validates (scan_basis, activation_basis] inside the writer
queue and changes the constraint to ready only when the rebuilt AVET covers
activation_basis. An implementation without pending-constraint enforcement
must block writes for that final validation pass. Validation of only the schema
fingerprint is not sufficient.
Backfill uses the existing index publisher and bypasses normal index-policy
pacing, as corium db request-index does. Apply reports the forced
publication's progress and does not mark readiness until the published root
carries both the target schema generation and readiness basis. A failure leaves
the attribute pending and unready. Queries use the correct AEVT fallback. The
ordinary interval, tail threshold, and deadline never decide correctness.
Protection changes reuse the forward-only model in
encryption.md, and are implemented.
:db/protection is part of the normalized desired model in both syntaxes and
reaches the desired digest, so a file that names a class is compared against
the installed timeline rather than reading as "no changes".
Applying one appends to the attribute's timeline at the committing basis
instead of rewriting it: protecting appends (t, class), unprotecting appends
(t, None). Nothing already stored is re-sealed, opened, or rewritten. That is
what keeps a value retractable by naming the bytes it was asserted as, and what
lets a keyless transactor still decide which forms a later retraction or
:db/cas operand may name. ever_protected therefore stays true forever, and
the attribute is permanently ineligible for index or unique coverage — even
after it is unprotected again.
:db/protection cannot coexist with :db/index, :db/unique, or
:db.type/ref, because ciphertext order is not value order. The planner
enforces this with the protection-conflict blocked reason, on a brand-new
attribute as well as on a change, so a declaration that CreateDatabase would
reject is not silently accepted as an update. The operator drops :db/index
and :db/unique from the same declaration; the plan says so. Every protection
change is a high-risk validate-reindex step requiring
protection-forward-only, and the plan reports plainly that lookup refs and
value-ordered reads through the attribute stop working.
A schema update can point an attribute at an installed class. It cannot install a class definition — the key identity, algorithm, scope, padding, and policies are create-time work — and a file that defines one says so in the plan notes rather than having it quietly ignored.
An immutable database cannot make an attribute disappear from history. Schema removal therefore means retirement:
Changing an attribute's value type in place has the same historical problem and
an additional transactional one: old retractions and CAS operands retain their
old type. Direct type mutation is therefore destructive, not rewrite, and
--allow rewrite can never enable it. The planner rejects that change and emits
a shadow-attribute recipe instead:
The migration may be automated later, but its conversion policy cannot be
inferred from the two type names. long -> string, for example, still needs a
declared formatting contract, while double -> long needs rounding and range
rules.
Corium will install the schema vocabulary itself in the reserved db partition:
:db/ident, :db/valueType, :db/cardinality, :db/unique, :db/index,
:db/isComponent, :db/noHistory, :db/doc, :db/protection when enabled,
and a retirement marker. Attributes supplied at database creation form an
immutable pre-basis schema seed. The seed has no transaction id, is not a
datom view, and is never returned by datoms, since, history, or
tx-range. Basis 0 therefore keeps its existing meaning: as-of 0 contains no
facts and since 0 contains every transacted fact. Later attribute metadata
changes travel as ordinary schema datoms in the transaction log.
Db::with_transaction_at derives the next immutable Schema and Idents
before it indexes user datoms from the same transaction. Schema validation
rejects illegal transitions, protects engine attributes, and makes the result
independent of datom input order. A transaction that both installs an attribute
and uses it is legal only because schema effects are derived first.
The authority for schema at basis t is the immutable pre-basis seed plus
schema datoms through t. The metadata root stores that seed and can also carry
a recoverable current-schema snapshot/cache with the generation and basis it
represents. Replay of the log tail advances the cache exactly as replay advances
data. Existing databases already have the information needed for the seed in
their creation metadata. The compatibility upgrade labels and preserves it
rather than inventing a transaction. The operation is deterministic and
idempotent and preserves attribute ids, basis 0, and every application
transaction number.
Peers receive schema changes in tx reports. Applying such a report replaces the
schema cache, invalidates affected in-memory index folds and planner statistics,
and publishes the new schema generation with the resulting Db. The next
protocol version makes the handshake schema snapshot explicitly effective at
schema_basis_t = SubscribeRequest.from_basis_t, with its generation and the
server's target generation carried separately. A cold peer subscribing from 0
receives the pre-basis seed in that snapshot. It is not backfilled as a t = 0
report. Reports with t > from_basis_t then advance data and schema together,
so a reconnect never applies an older data transaction against the server's
newest schema.
Db retains enough schema history to derive the effective schema and ident map
for as-of views. This history includes the protection timeline from
encryption.md. Retirement therefore does
not hide an attribute from an older basis. A future explicit rename can resolve
the ident that was active at that basis. Current and historical index readiness
are tracked independently when their coverage differs.
Published index roots carry the schema generation used to build them. If a root's generation is behind the current schema, uncovered orders remain usable where their semantics did not change, while affected orders rebuild or fall back. A root must never claim AVET completeness for a newly indexed or unique attribute until backfill has completed.
The schema generation is a monotone database-local counter, separate from transaction basis. It advances once for a committed transaction containing one or more schema changes. The basis says when the change happened. The generation shows whether two otherwise different database values use the same schema.
corium-forms owns format-specific parsing and a normalized desired
attribute model that does not allocate entity ids. Database creation keeps
its positional allocation so embedded, WASM, authz bootstrap, and existing
fixtures remain reproducible. Dynamic updates allocate above the durable
maximum installed db-partition id.corium-core owns installed attribute state, retirement/readiness metadata,
the optional :db/doc value (not represented by Attribute today),
protection timelines, schema generations, and property-level change/plan
types shared across clients and the transactor.corium-db derives schema timelines from datoms, exposes fixed-basis impact
scans, and invalidates only the index/statistic folds affected by a schema
generation.corium-tx validates schema datoms and transition rules before ordinary
transaction validation. Retractions remain legal for retired attributes and
for historically valid value representations.corium-protocol adds versioned plan/apply messages and carries schema
generation/readiness in recovery and subscription state. Applying a plan is
an administrative catalog action, not an unrestricted transaction form. The
required protocol, public thin-client, and authorization changes are also
recorded in protocol.md,
the thin-client contract, and
auth.md.corium-transactor allocates db-partition ids, verifies plan preconditions in
the writer queue, appends the schema transaction, and coordinates final
activation with index jobs that force the existing publisher past normal
index-policy pacing. Read-only impact scans operate on an immutable Db
snapshot outside the commit lock.corium-peer applies schema tx reports in order and makes the installed
schema/impact planner available to local clients.corium-cli renders human and stable JSON plans, enforces acknowledgement
flags, and routes long steps to the operator service when configured.Planning is read-only:
--prune
mode. Hash that encoding as the plan digest.--json.Observations — counts, samples, and work estimates at the observed basis — are carried beside the logical plan as advisory review and audit data. They are not hashed into the plan digest. Otherwise one unrelated new datom can change a count and invalidate the additive plan that the drift rule intentionally keeps valid.
Apply submits the desired schema, plan digest, observed basis, installed-schema fingerprint, and explicit acknowledgements to a schema-update endpoint. The transactor recomputes the canonical logical digest from the submitted desired schema and current installed-schema fingerprint. It does not treat the digest as an opaque server-side token. It does not recompute advisory observations. It independently validates every safety-critical precondition under its single-writer commit queue. A changed schema fingerprint, a newly introduced constraint violation, or a change in execution class returns a stale/blocked plan error and changes nothing. Data-basis drift that preserves all preconditions is allowed. Thus, a busy database can add an attribute. Additive metadata changes commit in one schema transaction with transaction metadata recording the source digest, plan digest, observed basis, tool version, and requester.
Reindex and rewrite steps are jobs. Each step is idempotent and checkpointed by attribute and basis. The CLI may execute the initial local implementation, but the job contract matches the operator service in operator-service.md, so the same plan can later run in a durable operator process. Constraint activation is a final short, basis-fenced transaction after the job verifies its result.
Applying the same desired digest twice is a no-op when the installed properties already match. A partially completed plan is resumed by its step keys, not replayed blindly. A different desired digest creates a new plan.
The initial surface is intentionally narrow:
corium schema update <db> --schema <path>
[--prune] [--json]
[--detailed-exit-code]
[--apply --plan <digest>]
[--allow validate-reindex|rewrite]
[--ack <change-code>...]
[connection flags]
--apply, the command never writes.--apply requires the digest printed by the plan, preventing an unnoticed
plan/apply mismatch.--allow flag. Higher classes require their exact
allowance. destructive has no allowance because this command cannot run it.--ack. Allowing an execution class alone does not acknowledge its meaning.component-enable,
retire-live-attribute, unique-mode-change, no-history-enable, and
protection-forward-only. Both human and JSON plans print the exact code next
to every change that requires it.--prune requests retirement of absent installed attributes and is included
in the digest.--json is a versioned machine contract and includes stable change codes.
Scripts must not parse the human rendering.&& chains remain useful. --detailed-exit-code requests 0 for no change and
2 for changes planned. Parse, stale-plan, blocked, and apply failures exit 1
and carry stable JSON error codes when --json is set.--apply whose freshly computed plan contains no changes succeeds and
reports changed: false, whatever --plan names. Installing a change is
exactly what invalidates the digest that described it, so demanding the
digest here would make the command fail the second time it ran. An empty plan
writes nothing, so there is nothing for a digest to protect.schema is a deliberate top-level group rather than another db verb. Its
planned surface includes update, status, history, and job inspection.
corium db remains the group for catalog lifecycle and index-policy operations.
Example human plan:
database: people basis: 418 schema-generation: 3
desired: sha256:7c… plan: sha256:91…
ADDITIVE
+ :person/email string cardinality-one
~ :person/tags cardinality one -> many current datoms: 12,204
VALIDATE-REINDEX
~ :person/email unique none -> identity duplicate values: 0
AVET backfill: 0 current datoms
~ :person/address component false -> true live refs: 8,109
[ack: component-enable]
~ :person/ssn protection none -> :protect/pii current datoms: 8,109
[ack: protection-forward-only]
note: values written from this
basis onward are sealed; the
plaintext already stored keeps
its form
note: lookup refs and
value-ordered reads through this
attribute stop working
DESTRUCTIVE (blocked)
~ :person/age long -> string current datoms: 8,109
direct type mutation is not executable
rewrite recipe: add :person/age-text, copy explicitly, cut over, retire old
UNMANAGED
:legacy/import-id use --prune to retire
Planning requires inspect access to the database. Applying schema metadata
requires a new AlterSchema action, separate from ordinary Transact, so an
application writer cannot silently broaden its own schema. AlterSchema is an
Admin-class, database-scoped action with wire name alter-schema. The built-in
permission defaults therefore grant it only to database owners. Rewrite jobs
also require ordinary database-level Transact authority. Corium has no
attribute-scoped transact permission today. This design does not add this
permission. Hard deletion/excision, when designed, belongs to the operator
service's irreversible two-person approval path.
Every applied schema transaction records requester identity, desired and plan
digests, CLI/protocol version, execution class, and acknowledgements on the
transaction entity, under the :db.schemaUpdate/* attributes
(requester, desiredDigest, planDigest, observedBasis, tool, and the
cardinality-many class and ack). They are ordinary queryable attributes —
that is the point of keeping the audit trail in the database — but Transact
refuses them, so a transaction cannot claim to have been a schema update. The
requester is the authenticated principal, never a field the caller supplies.
Plans may contain samples for diagnosis but audit records store counts and
digests rather than application values.
Implemented. corium_forms::desired normalizes both syntaxes,
corium_db::impact runs the fixed-basis scans, corium_forms::planner
produces the plan, corium_core::migration owns the change/plan types and
digests, and corium-cli renders it. Operator documentation is in
operations.md.
Implemented. The vocabulary is bootstrapped, corium_db::schemadatoms
derives basis-versioned schema and ident maps from schema datoms, and
corium_forms::apply compiles a reviewed plan into them. Updates allocate ids
above the durable maximum while creation keeps its positional allocation.
Catalog.AlterSchema carries the change, corium_transactor::node verifies it
under the writer queue, and --apply drives it.
The schema generation is derived per database value and travels implicitly:
peers reach it by applying tx reports, so a live peer converges on the new
schema without a protocol change. Carrying the generation explicitly through
recovery roots and the subscription handshake — so a peer that cannot
understand a generation fails the version check rather than applying data under
stale rules — is not done, and neither is the handshake's
schema_basis_t = from_basis_t contract. A cold peer still receives the
current schema snapshot rather than the one effective at its subscription
basis.
Partly implemented. Index, uniqueness, component, no-history,
documentation, and cardinality changes all apply, and the exact duplicate and
cardinality-conflict scans block the plans they must. The permanent
ever-protected prohibition on later index/unique coverage is enforced.
The two-stage requested/ready index state is not implemented. It is not yet
observable: a peer holds the whole database in memory and rebuilds a covering
index from the log whenever the schema generation advances, so coverage is
total the moment the change commits, and the uniqueness validator has a correct
AEVT fallback in Db::lookup. That equivalence is a property of the current
in-memory index, not of the design. Published segment roots do not yet carry
the schema generation they were built under, so the readiness basis, the
pending-constraint enforcement, and the forced publication past index-policy
pacing all remain to build alongside them.
Partly implemented. Retirement applies: --prune plans it, :db/retired
records it, and corium-tx refuses new assertions while keeping every existing
fact readable and retractable.
Checkpointed current-fact rewrite jobs, the explicit conversion and
conflict-resolution interface, and routing long jobs through the operator
service are not built. rewrite plans are still refused at apply.
--prune, protection timelines, engine attributes, stable acknowledgement
codes, deterministic ordering, and plan digests.Can you improve this documentation? These fine people already did:
Casey Marshall & ClaudeEdit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |