konserve-lmdb.fork)Experimental. ZFS-only for now, and the API may change. A second copy-on-write backend (btrfs) would introduce a protocol here; until one exists the surface stays concrete.
A fork duplicates a whole konserve-lmdb store — every key, and for a datahike backend its whole index and history — in constant time. The copies share disk blocks until they diverge, so a fork is the cheap way to spin up a read replica, a throwaway experiment, or a physical backup of a live store. It is filesystem copy-on-write (a ZFS clone today), not a logical copy: nothing is re-encoded, and the store bytes are never walked.
konserve-lmdb has three, and they are orthogonal — pick per use case:
| Mechanism | Granularity | Cost | Portable? |
|---|---|---|---|
mode-B versions (:versioned? true) | per key, in one store | a version per write | yes — plain LMDB bytes |
yggdrasil durable (logical CRDT branch) | a named branch of values | a commit per change | yes — content-addressed DAG |
| physical fork (this namespace) | the whole store at once | O(1), COW-shared | no — tied to the filesystem |
A fork versions the physical bytes; the other two version values inside a store. Forking a versioned store forks its versions too — the mechanisms compose.
LMDB survives a filesystem snapshot even while open. Its dual meta-page plus
fsync-on-commit mean a snapshot always lands on a committed B-tree root, so
forking a running store captures its last committed state, consistently. If
a specific write must be included, sync (or close) it first.
Of the ZFS operations a fork uses, only mount(2) ever needs elevation.
snapshot / clone / create / destroy / list are all delegable and run
unprivileged once an admin grants them with zfs allow (see below). So the
whole privilege question reduces to: how does this process get a dataset
mounted? That is the backend's :mount mode.
On Linux, ZFS delegation does not cover
mount(it needsCAP_SYS_ADMIN); on FreeBSD/illumos it can. Hence the three modes.
zfs-backend(require '[konserve-lmdb.fork :as fork])
(def backend
(fork/zfs-backend
{:pool "tank" ; ZFS pool
:prefix "klmdb" ; datasets live at tank/klmdb/<name>
:mount-base "/klmdb" ; and mount at /klmdb/<name>
:mount :direct})) ; how mount(2) is obtained (see below)
:pool, :prefix and :mount-base are all required. A store name is a
single path segment ([A-Za-z0-9][A-Za-z0-9_-]*) — it can never escape the
pool/prefix namespace or inject a flag.
:mount modes| Mode | What it does | Use when |
|---|---|---|
:direct (default) | the process runs zfs mount itself | it HAS the rights — running as root, or ZFS delegation on a platform where mount is delegable (FreeBSD/illumos) |
:sudo | sudo -n zfs … for every call, plus sudo -n chown to hand the fresh mount back to the caller | you have passwordless sudo for zfs and chown |
{:helper "PATH"} | a narrow, prefix-guarded sudo helper mounts AND chowns; create/clone/destroy still run directly (delegated) | an unprivileged process with a vetted broker — the recommended production shape |
As an admin, once:
# a parent dataset to hold all forkable stores
sudo zfs create -o mountpoint=/klmdb tank/klmdb
Every store <name> becomes tank/klmdb/<name>, mounting at /klmdb/<name>.
Grant the runtime user everything except mount, scoped to the sandbox:
sudo zfs allow -u appuser \
create,clone,snapshot,destroy,mount,hold,release,mountpoint \
tank/klmdb
(mount is listed so ZFS permits it where the platform allows delegated
mounts; on Linux the kernel still requires root for the mount syscall itself,
which is what the helper or sudo covers.)
Simplest — :direct, running as root or on FreeBSD/illumos. Nothing more to
do; zfs-backend defaults to it.
Passwordless sudo — :sudo. Grant appuser NOPASSWD for zfs and chown,
then pass :mount :sudo. Broad, but simple.
Recommended — a guarded helper {:helper "…"}. A tiny root-owned script that
can do exactly two things (mount / umount) to exactly one subtree, then hand
the fresh mount root back to the caller. Because the sandbox is hard-coded in the
script (never an argument), a compromised caller cannot name tank/ROOT, a
sibling, or a snapshot through it. A template:
#!/usr/bin/env bash
# klmdb-zfs-mount.sh — mount/umount ONE segment under a fixed sandbox, then
# chown the mount root to the caller. The only operations that need root.
set -euo pipefail
readonly SANDBOX='tank/klmdb' # <-- EDIT: your pool/prefix. Hard-coded, never an argument.
readonly MOUNT_BASE='/klmdb' # <-- EDIT: your mount-base. Mountpoints must sit under here.
readonly op="${1-}" child="${2-}"
case "$op" in mount|umount) ;; *)
echo "op must be 'mount' or 'umount', got '${op}'" >&2; exit 2 ;;
esac
# The whole safety rail: one plain segment, no slash / '..' / '@' / leading dash.
if [[ ! "$child" =~ ^[A-Za-z0-9][A-Za-z0-9_-]*$ ]]; then
echo "refusing child '${child}'" >&2; exit 2
fi
readonly ds="${SANDBOX}/${child}"
if [[ "$ds" != "${SANDBOX}/${child}" || "$ds" == "$SANDBOX" ]]; then
echo "composed target '${ds}' failed the sandbox check" >&2; exit 2
fi
if [[ "$op" == umount ]]; then
exec zfs unmount "$ds"
fi
# mount, then hand the (root-owned) mount root back to the caller so an
# unprivileged process can write its data.mdb into the clone.
zfs mount "$ds"
mp="$(zfs get -H -o value mountpoint "$ds")"
[[ "$mp" == "${MOUNT_BASE}/"* ]] || { echo "mountpoint '${mp}' outside ${MOUNT_BASE}/" >&2; exit 2; }
owner="${SUDO_USER:-}"
[[ -n "$owner" && "$owner" != root ]] || { echo "SUDO_USER unset or root" >&2; exit 2; }
exec chown "${owner}:${owner}" "$mp" # non-recursive on purpose: only the mount root
Install it root-owned, non-writable by the caller, and NOPASSWD for that one path:
sudo chown root:root /opt/klmdb/klmdb-zfs-mount.sh
sudo chmod 755 /opt/klmdb/klmdb-zfs-mount.sh
echo 'appuser ALL=(root) NOPASSWD: /opt/klmdb/klmdb-zfs-mount.sh' \
| sudo tee /etc/sudoers.d/klmdb
sudo chmod 440 /etc/sudoers.d/klmdb
Then:
(def backend
(fork/zfs-backend {:pool "tank" :prefix "klmdb" :mount-base "/klmdb"
:mount {:helper "/opt/klmdb/klmdb-zfs-mount.sh"}}))
A working copy of this helper (hard-coded to the project's own lab sandbox) lives
at bench/klmdb-zfs-mount.sh.
(require '[konserve-lmdb.fork :as fork]
'[konserve-lmdb.store :as s]
'[konserve.core :as k])
;; provision a fresh empty store and connect it
(def pa (fork/init-store! backend "orders")) ; => "/klmdb/orders"
(def sa (s/connect-store pa))
(k/assoc sa :x 1 {:sync? true})
(s/release-store sa)
;; O(1) physical fork into a new, independent store
(def pb (fork/fork! backend "orders" "orders-exp")) ; => "/klmdb/orders-exp"
(def sb (s/connect-store pb))
(k/get sb :x nil {:sync? true}) ; => 1 (sees the source's data)
(k/assoc sb :y 2 {:sync? true}) ; diverges; the source never sees :y
(s/release-store sb)
| Function | Purpose |
|---|---|
(init-store! backend name) | provision a fresh empty dataset, return its path |
(fork! backend src dst) | O(1) COW fork of src's latest state into new store dst |
(fork! backend src tag dst) | fork from a specific snapshot tag instead of latest |
(snapshot! backend name tag) | snapshot the current committed state as tag |
(snapshots backend name) | snapshot tags, oldest first |
(destroy! backend name) | unmount and destroy a store and its own snapshots |
(destroy-snapshot! backend name tag) | destroy one snapshot, keep the store |
(stores backend) | names of all stores under the backend's namespace |
fork! leaves its snapshot on the source (a fork is a ZFS clone, which pins its
origin snapshot). Connect the returned path with konserve-lmdb.store yourself —
the fork namespace only manages datasets, not store lifecycles.
destroy! on the origin will fail while a
clone of it exists.konserve-lmdb.store/compact! for that.konserve-lmdb.yggdrasil — maps yggdrasil's clone=branch model onto these
forks, turning a store into a branchable, durable history.konserve-lmdb.fork namespace docstring, for the authoritative reference.Can you improve this documentation?Edit on GitHub
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |