Datahike uses persistent data structures that enable structural sharing—each update creates a new version efficiently by reusing unchanged parts. This allows time-travel queries and git-like versioning, but storage grows over time as old snapshots accumulate.
Garbage collection removes old database snapshots from storage while preserving current branch heads.
Garbage collection and data purging are different operations:
The two compose for actual erasure:
purge produces a new commit whose indices no longer reach the targeted datoms.(d/commit-as-db conn <pre-purge-uuid>) still sees the purged data.d/gc-storage with a grace-period cutoff old enough to drop the pre-purge commit physically evicts those nodes from konserve.So the erasure recipe is purge + cutoff-GC, not purge alone. Plain d/gc-storage (no cutoff) only reclaims storage from deleted branches and leaves intermediate commits intact — useful as routine maintenance, but not as the eviction step for erasure.
For multi-branch databases, purge on every branch that holds the datom; for how secondary indices participate, see Secondary indices: purge propagation.
GC whitelists all current branches and marks snapshots as reachable based on a grace period. Snapshots older than the grace period are deleted from storage, but branch heads are always retained regardless of age.
(require '[datahike.api :as d]
'[superv.async :refer [<?? S]])
;; Remove only deleted branches, keep all snapshots
(<?? S (d/gc-storage conn))
;; => #{...} ; set of deleted storage blobs
Running without a date removes only deleted branches—all snapshots on active branches are preserved. This is safe to run anytime and reclaims storage from old experimental branches.
Note: Returns a core.async channel. Use <?? to block, or run without it for background execution. GC never blocks transactions or reads.
Run GC where the writers are. A commit writes every value the new head references and only then flips the head — so for the duration of that sequence those objects exist in the store and nothing yet names them. A collector in the same process knows a commit is in flight and leaves them alone (datahike.gc-guard). A collector in another process cannot know: its view of "what is in flight" is empty because its heap is idle, not because the store is quiet.
With a single exclusive writer (:writer {:backend :self :writer-ownership :exclusive}) that is the whole story: d/gc-storage is a writer operation, so it already runs in the right place.
With shared writers (:writer-ownership :shared, the default) several processes may commit to the same branch. Head fencing keeps them from losing each other's commits, but a fence protects the pointer, not the values: a commit in flight in another process is invisible to the collector, and its objects are on disk reachable from nothing. The same is true of a cron sidecar collecting a store it never writes. In both cases the only protection is the sweep floor:
(d/gc-storage conn (java.util.Date. 0) {:min-age-ms (* 15 60 1000)})
:min-age-ms spares anything written more recently than that, whatever the mark says. Size it above the longest window between "first value written" and "head flipped" any of your writers can have — a writer that awaits its transacts has that window closed when the call returns, so one request's duration is the bound — plus the largest clock difference between your processes: the :last-write stamps the sweep compares against come from each writer's own clock, so a writer twenty minutes behind the collector looks twenty minutes older than it is. A suspended process (a frozen Lambda resuming mid-commit) can exceed any bound you pick. The price of a generous value is only delayed reclamation; the price of a small one is a dangling head.
Datahike's Distributed Index Space allows readers to access storage directly without coordination. This is powerful for scalability but means long-running processes might read from old snapshots for hours.
Examples of long-running readers:
The grace period ensures these readers don't encounter missing data. Snapshots created after the grace period date are kept; older ones are deleted.
(require '[datahike.api :as d])
;; Keep last 7 days of snapshots
(let [seven-days-ago (java.util.Date. (- (System/currentTimeMillis)
(* 7 24 60 60 1000)))]
(<?? S (d/gc-storage conn seven-days-ago)))
;; Keep last 30 days (common for compliance)
(let [thirty-days-ago (java.util.Date. (- (System/currentTimeMillis)
(* 30 24 60 60 1000)))]
(<?? S (d/gc-storage conn thirty-days-ago)))
;; Keep last 24 hours (for fast-moving data)
(let [yesterday (java.util.Date. (- (System/currentTimeMillis)
(* 24 60 60 1000)))]
(<?? S (d/gc-storage conn yesterday)))
Choosing a grace period:
Branch heads are always kept regardless of the grace period—only intermediate snapshots are removed.
⚠️ EXPERIMENTAL FEATURE
Online GC automatically deletes freed index nodes during transaction commits, preventing garbage accumulation during bulk imports and high-write workloads.
Online GC is currently an experimental feature. While it has been tested extensively in Clojure/JVM and includes safety mechanisms for multi-branch databases, use with caution in production. We recommend:
- Thorough testing in your specific use case before production deployment
- Monitoring freed address counts to verify expected behavior
- Using it primarily for bulk imports and high-write workloads where it's most beneficial
- ClojureScript: Online GC functionality is available in CLJS but has not been tested in big bulk loads yet. JVM testing is more comprehensive.
- Reporting any issues at https://github.com/replikativ/datahike/issues
Online GC has two hard requirements: a single branch, and
:diff-buf-size 0. Both are enforced — the diff-buf combination is refused at connect, and both are skipped inside the GC itself. Use offline GC (d/gc-storage) otherwise; it derives reachability itself instead of trusting the freed-address hint, so it is unaffected.Note the multi-branch restriction is not because structural sharing is confined to branches — it is not. Nodes are shared between any two versions, including two versions of the same branch. See Why diff-buf is excluded below.
When PSS (Persistent Sorted Set) index trees are modified during transactions, old index nodes become unreachable. Online GC tracks these freed addresses with timestamps and deletes them incrementally:
markFreed() for each replaced index nodeKey benefits:
Enable online GC in your database config:
;; For bulk imports (no concurrent readers, single-branch)
;; See "Address Recycling" section below for details
{:online-gc {:enabled? true
:grace-period-ms 0 ;; Recycle immediately
:max-batch 10000} ;; Large batches for efficiency
:crypto-hash? false} ;; Required for address recycling
;; For production (concurrent readers)
{:online-gc {:enabled? true
:grace-period-ms 300000 ;; 5 minutes
:max-batch 1000}} ;; Smaller batches
;; Disabled (default)
{:online-gc {:enabled? false}}
Configuration options:
:enabled? - Enable/disable online GC (default: false):grace-period-ms - Minimum age in milliseconds before deletion (default: 60000 = 1 minute):max-batch - Maximum addresses to delete per commit (default: 1000):sync? - Synchronous deletion (always false inside commits for async operation)For production systems, run GC in a background thread instead of blocking commits:
(require '[datahike.online-gc :as online-gc])
;; Start background GC
(def stop-ch (online-gc/start-background-gc!
(:store @conn)
{:grace-period-ms 60000 ;; 1 minute
:interval-ms 10000 ;; Run every 10 seconds
:max-batch 1000}))
;; Later, stop background GC
(clojure.core.async/close! stop-ch)
Background mode advantages:
⚠️ EXPERIMENTAL FEATURE
Address recycling is an experimental optimization. It has been designed with safety checks (multi-branch detection, grace periods), but should be thoroughly tested in your environment before production use.
Online GC includes address recycling—freed addresses are reused for new index nodes instead of being deleted from storage. This optimization is particularly powerful for bulk imports.
How it works:
Benefits:
:grace-period-ms 0, recycling happens immediatelySafety limitations:
Address recycling is ONLY safe for:
Online GC is automatically disabled when:
:diff-buf-size is non-zero (refused at connect; also skipped with a warning inside the
GC if reached via :allow-unsafe-config or a direct online-gc! call).
Reason: see Why diff-buf is excluded below:crypto-hash? true with recycling (falls back to deletion mode)Online GC reclaims blobs from the markFreed stream that persistent-sorted-set emits.
That stream is documented by pss as a hint, not a reachability claim — the consumer
must establish for itself that no live version needs an address.
Without diff-buf the hint is reliable in practice: a changed child is always rewritten to a new address, so the old one belonged solely to the version being superseded.
With diff-buf it is not. A parent no longer says "child i is the blob at A"; it says "child i is the blob at anchor A plus this diff". Two versions can therefore name the same anchor with different diffs, and neither owns it. Storing one of them may flush that child — write it out whole and free the anchor — which is correct for the version being stored and wrong for the other. The useful distinction is that freeing on supersession is safe while freeing on re-representation is not, and a flush is re-representation: the same elements, written differently.
So the stream is sound exactly when the commit history is linear — every version stored before the next is derived from it. Measured in pss against a backend that acts on the callback:
| shape | result |
|---|---|
| linear | publication closure held in 432/432 cells, every budget |
| ancestor then descendant (descendant derived while ancestor unstored) | 25 read failures / 768 trials at budget ≤ 4; clean at ≥ 8 |
:diff-buf-size 0 | 864/864 clean, no premature free in any shape |
Two mitigations that do not work, so they are not attempted: retaining "the last N
images" fails for every N, because the freed blob is not reachable from the immediately
preceding image and the required depth grows without bound; and limiting the number of
branches does not help, because the hazardous shape lives inside a single lineage
(force-branch! twice on one branch reproduces it while (count branches) stays 1).
Datahike enforces the simpler, mechanically checkable rule — :diff-buf-size 0 — rather
than asking users to reason about linearity. That is also strictly stronger: every free
that could be premature happens under diff-buf, so excluding it closes the class.
Tracked in datahike#951 — see the issue
for the underlying design question of whether incremental reclamation should derive its
free set from reachability instead of a producer hint. pss's own statement of the contract
is in IStorage.markFreed.
For maximum performance during bulk imports where no concurrent readers exist:
;; Optimal bulk import configuration
{:online-gc {:enabled? true
:grace-period-ms 0 ;; Recycle immediately (no readers)
:max-batch 10000} ;; Large batch (only for delete fallback)
:crypto-hash? false ;; Required for recycling
:branch :db} ;; Single branch only
;; Example bulk import
(let [cfg {:store {:backend :file :path "/data/bulk-import"}
:online-gc {:enabled? true :grace-period-ms 0}
:crypto-hash? false}
conn (d/connect cfg)]
;; Import millions of entities
(doseq [batch entity-batches]
(d/transact conn batch))
;; Storage stays bounded - addresses are recycled
(d/release conn))
Bulk import best practices:
:grace-period-ms 0 (no concurrent readers to protect):crypto-hash? false (enables address recycling):branch :db):max-batch for efficiency (only affects delete fallback)Verifying address recycling:
"Online GC: recycling N addresses to freelist""Online GC: skipped (multi-branch detected)", ensure single branch
(multi-branch databases require offline GC instead)Online GC (incremental):
Offline GC (d/gc-storage):
:diff-buf-size (online GC doesn't work in either case)Use both: Online GC for incremental cleanup during single-branch writes, offline GC for periodic deep cleaning and all multi-branch scenarios.
With online GC enabled, garbage collection becomes largely automatic during normal operation. Manual d/gc-storage runs are only needed for:
GC removes:
GC preserves:
Remember: Actual erasure (GDPR / HIPAA / CCPA) requires purging followed by a cutoff d/gc-storage sweep. Purge alone leaves the pre-purge commit reachable; GC alone doesn't delete data on a live snapshot.
Experimental. Everything above rests on the collector seeing what is in flight, which it can only do inside its own process. Some work is too long for that to be enough, wherever it runs: a secondary-index backfill scanning a snapshot for an hour, a bulk import building trees for hours before it publishes them. For those, datahike.gc-roots lets a process persist a root in the store — a record the mark walks in addition to the branch heads, so a collector in any process keeps what it names.
A root protects only what a record names. That is the whole idea, and the whole limit:
:pin — a copy of a commit record with its parents removed. Keeps that commit's trees, schema, secondary-index key-maps and the blobs its datoms name, and nothing older. For a long reader of an old snapshot.:checkpoint — a synthetic record in commit shape whose fields name partial state: the trees of a build in progress. For a long builder; republish it as the build advances.:ref — a commit record with its parents kept, so its ancestry is retained under the same remove-before gating as a branch. For durable references to old commits; permanent unless given a TTL.(require '[datahike.gc-roots :as roots])
(def id (<?? S (roots/pin! (d/db conn) {:note "report job" :ttl-ms (* 2 60 60 1000)})))
;; … hours of work against that snapshot …
(<?? S (roots/release! (d/db conn) id))
Roots carry a lease. A holder that dies must not pin forever, so an entry expires; the holder renews it (renew!, or start-renewal! for a background loop) at a fraction of its TTL, and the collector reaps an entry once it is past expiry by its own TTL again. A holder that finds its entry gone at renewal — reaped, or deleted by an older Datahike whose sweep does not know the registry — gets :gc/root-lost and must abandon what the root was protecting; assert-live! is the same check for the moment before publishing. Timestamps are wall-clock and only decide expiry; the sweep cutoff is unaffected.
What roots do not do: cover the milliseconds between "values written" and "record published" — no record exists yet, so that stays with the guard and the floor above. And a root pins a record older than the head, which makes online GC's freed-address hints unsound for the same reason multiple branches do; online GC pauses while any root exists.
With no roots declared, nothing changes: the registry key is never written and the mark is exactly what it was.
Can you improve this documentation?Edit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |