Datahike uses persistent data structures that enable structural sharing—each update creates a new version efficiently by reusing unchanged parts. This allows time-travel queries and git-like versioning, but storage grows over time as old snapshots accumulate.
Garbage collection removes old database snapshots from storage while preserving current branch heads.
Garbage collection and data purging are different operations:
The two compose for actual erasure:
purge produces a new commit whose indices no longer reach the targeted datoms.(d/commit-as-db conn <pre-purge-uuid>) still sees the purged data.d/gc-storage with a grace-period cutoff old enough to drop the pre-purge commit physically evicts those nodes from konserve.So the erasure recipe is purge + cutoff-GC, not purge alone. Plain d/gc-storage (no cutoff) only reclaims storage from deleted branches and leaves intermediate commits intact — useful as routine maintenance, but not as the eviction step for erasure.
For multi-branch databases, purge on every branch that holds the datom; for how secondary indices participate, see Secondary indices: purge propagation.
GC whitelists all current branches and marks snapshots as reachable based on a grace period. Snapshots older than the grace period are deleted from storage, but branch heads are always retained regardless of age.
(require '[datahike.api :as d]
'[superv.async :refer [<?? S]])
;; Remove only deleted branches, keep all snapshots
(<?? S (d/gc-storage conn))
;; => #{...} ; set of deleted storage blobs
Running without a date removes only deleted branches—all snapshots on active branches are preserved. This is safe to run anytime and reclaims storage from old experimental branches.
Note: Returns a core.async channel. Use <?? to block, or run without it for background execution. GC never blocks transactions or reads.
Run GC in the process that writes. d/gc-storage is a writer operation and already runs there, so in the normal case you get this for free — but it is worth stating, because "collect from a cron job during the quiet hours" is a tempting shape and it is the wrong one.
This follows from Datahike's writer model rather than from anything about GC:
All writers for a database run in one JVM. A connection owns its branch head and serializes commits through it. Different branches of the same database may each have their own writer, but they belong in the same process — a database's writers coordinate in memory, not through the store. Readers are unconstrained: any number of them, in any number of processes, anywhere.
The collector belongs on the writers' side of that line because it has to know what they are currently writing. A commit writes every value the new head references and only then flips the head — so for the duration of that sequence, those objects exist in the store and nothing yet names them. A collector in the same process knows a commit is in flight and leaves them alone. A collector in another process cannot know, and Datahike cannot tell you it doesn't: a second process looks like a writer too.
(Cross-process writers are outside the model for a more basic reason as well — there is no head fencing yet, so two writers on a branch can lose each other's commits regardless of GC. See #878.)
Datahike's Distributed Index Space allows readers to access storage directly without coordination. This is powerful for scalability but means long-running processes might read from old snapshots for hours.
Examples of long-running readers:
The grace period ensures these readers don't encounter missing data. Snapshots created after the grace period date are kept; older ones are deleted.
(require '[datahike.api :as d])
;; Keep last 7 days of snapshots
(let [seven-days-ago (java.util.Date. (- (System/currentTimeMillis)
(* 7 24 60 60 1000)))]
(<?? S (d/gc-storage conn seven-days-ago)))
;; Keep last 30 days (common for compliance)
(let [thirty-days-ago (java.util.Date. (- (System/currentTimeMillis)
(* 30 24 60 60 1000)))]
(<?? S (d/gc-storage conn thirty-days-ago)))
;; Keep last 24 hours (for fast-moving data)
(let [yesterday (java.util.Date. (- (System/currentTimeMillis)
(* 24 60 60 1000)))]
(<?? S (d/gc-storage conn yesterday)))
Choosing a grace period:
Branch heads are always kept regardless of the grace period—only intermediate snapshots are removed.
⚠️ EXPERIMENTAL FEATURE
Online GC automatically deletes freed index nodes during transaction commits, preventing garbage accumulation during bulk imports and high-write workloads.
Online GC is currently an experimental feature. While it has been tested extensively in Clojure/JVM and includes safety mechanisms for multi-branch databases, use with caution in production. We recommend:
- Thorough testing in your specific use case before production deployment
- Monitoring freed address counts to verify expected behavior
- Using it primarily for bulk imports and high-write workloads where it's most beneficial
- ClojureScript: Online GC functionality is available in CLJS but has not been tested in big bulk loads yet. JVM testing is more comprehensive.
- Reporting any issues at https://github.com/replikativ/datahike/issues
Online GC has two hard requirements: a single branch, and
:diff-buf-size 0. Both are enforced — the diff-buf combination is refused at connect, and both are skipped inside the GC itself. Use offline GC (d/gc-storage) otherwise; it derives reachability itself instead of trusting the freed-address hint, so it is unaffected.Note the multi-branch restriction is not because structural sharing is confined to branches — it is not. Nodes are shared between any two versions, including two versions of the same branch. See Why diff-buf is excluded below.
When PSS (Persistent Sorted Set) index trees are modified during transactions, old index nodes become unreachable. Online GC tracks these freed addresses with timestamps and deletes them incrementally:
markFreed() for each replaced index nodeKey benefits:
Enable online GC in your database config:
;; For bulk imports (no concurrent readers, single-branch)
;; See "Address Recycling" section below for details
{:online-gc {:enabled? true
:grace-period-ms 0 ;; Recycle immediately
:max-batch 10000} ;; Large batches for efficiency
:crypto-hash? false} ;; Required for address recycling
;; For production (concurrent readers)
{:online-gc {:enabled? true
:grace-period-ms 300000 ;; 5 minutes
:max-batch 1000}} ;; Smaller batches
;; Disabled (default)
{:online-gc {:enabled? false}}
Configuration options:
:enabled? - Enable/disable online GC (default: false):grace-period-ms - Minimum age in milliseconds before deletion (default: 60000 = 1 minute):max-batch - Maximum addresses to delete per commit (default: 1000):sync? - Synchronous deletion (always false inside commits for async operation)For production systems, run GC in a background thread instead of blocking commits:
(require '[datahike.online-gc :as online-gc])
;; Start background GC
(def stop-ch (online-gc/start-background-gc!
(:store @conn)
{:grace-period-ms 60000 ;; 1 minute
:interval-ms 10000 ;; Run every 10 seconds
:max-batch 1000}))
;; Later, stop background GC
(clojure.core.async/close! stop-ch)
Background mode advantages:
⚠️ EXPERIMENTAL FEATURE
Address recycling is an experimental optimization. It has been designed with safety checks (multi-branch detection, grace periods), but should be thoroughly tested in your environment before production use.
Online GC includes address recycling—freed addresses are reused for new index nodes instead of being deleted from storage. This optimization is particularly powerful for bulk imports.
How it works:
Benefits:
:grace-period-ms 0, recycling happens immediatelySafety limitations:
Address recycling is ONLY safe for:
Online GC is automatically disabled when:
:diff-buf-size is non-zero (refused at connect; also skipped with a warning inside the
GC if reached via :allow-unsafe-config or a direct online-gc! call).
Reason: see Why diff-buf is excluded below:crypto-hash? true with recycling (falls back to deletion mode)Online GC reclaims blobs from the markFreed stream that persistent-sorted-set emits.
That stream is documented by pss as a hint, not a reachability claim — the consumer
must establish for itself that no live version needs an address.
Without diff-buf the hint is reliable in practice: a changed child is always rewritten to a new address, so the old one belonged solely to the version being superseded.
With diff-buf it is not. A parent no longer says "child i is the blob at A"; it says "child i is the blob at anchor A plus this diff". Two versions can therefore name the same anchor with different diffs, and neither owns it. Storing one of them may flush that child — write it out whole and free the anchor — which is correct for the version being stored and wrong for the other. The useful distinction is that freeing on supersession is safe while freeing on re-representation is not, and a flush is re-representation: the same elements, written differently.
So the stream is sound exactly when the commit history is linear — every version stored before the next is derived from it. Measured in pss against a backend that acts on the callback:
| shape | result |
|---|---|
| linear | publication closure held in 432/432 cells, every budget |
| ancestor then descendant (descendant derived while ancestor unstored) | 25 read failures / 768 trials at budget ≤ 4; clean at ≥ 8 |
:diff-buf-size 0 | 864/864 clean, no premature free in any shape |
Two mitigations that do not work, so they are not attempted: retaining "the last N
images" fails for every N, because the freed blob is not reachable from the immediately
preceding image and the required depth grows without bound; and limiting the number of
branches does not help, because the hazardous shape lives inside a single lineage
(force-branch! twice on one branch reproduces it while (count branches) stays 1).
Datahike enforces the simpler, mechanically checkable rule — :diff-buf-size 0 — rather
than asking users to reason about linearity. That is also strictly stronger: every free
that could be premature happens under diff-buf, so excluding it closes the class.
Tracked in datahike#951 — see the issue
for the underlying design question of whether incremental reclamation should derive its
free set from reachability instead of a producer hint. pss's own statement of the contract
is in IStorage.markFreed.
For maximum performance during bulk imports where no concurrent readers exist:
;; Optimal bulk import configuration
{:online-gc {:enabled? true
:grace-period-ms 0 ;; Recycle immediately (no readers)
:max-batch 10000} ;; Large batch (only for delete fallback)
:crypto-hash? false ;; Required for recycling
:branch :db} ;; Single branch only
;; Example bulk import
(let [cfg {:store {:backend :file :path "/data/bulk-import"}
:online-gc {:enabled? true :grace-period-ms 0}
:crypto-hash? false}
conn (d/connect cfg)]
;; Import millions of entities
(doseq [batch entity-batches]
(d/transact conn batch))
;; Storage stays bounded - addresses are recycled
(d/release conn))
Bulk import best practices:
:grace-period-ms 0 (no concurrent readers to protect):crypto-hash? false (enables address recycling):branch :db):max-batch for efficiency (only affects delete fallback)Verifying address recycling:
"Online GC: recycling N addresses to freelist""Online GC: skipped (multi-branch detected)", ensure single branch
(multi-branch databases require offline GC instead)Online GC (incremental):
Offline GC (d/gc-storage):
:diff-buf-size (online GC doesn't work in either case)Use both: Online GC for incremental cleanup during single-branch writes, offline GC for periodic deep cleaning and all multi-branch scenarios.
With online GC enabled, garbage collection becomes largely automatic during normal operation. Manual d/gc-storage runs are only needed for:
GC removes:
GC preserves:
Remember: Actual erasure (GDPR / HIPAA / CCPA) requires purging followed by a cutoff d/gc-storage sweep. Purge alone leaves the pre-purge commit reachable; GC alone doesn't delete data on a live snapshot.
Can you improve this documentation?Edit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |