Space reclamation for a file-backed SQLite store.
SQLite never hands the file back on its own: auto_vacuum is off, so pages a
delete or a DROP COLUMN frees go on the FREELIST and are only reused by
later writes. That is the right default — reuse costs nothing — but a one-off
bulk reclaim (retiring a column across a whole transcript) leaves hundreds of
megabytes the file keeps until someone runs VACUUM.
maybe-vacuum! is that someone, on the same fortnight window as
foundation.housekeeping retention: at most once per vacuum-interval-days
per store, and only when the freelist is worth the rewrite — at least
vacuum-min-free-bytes AND vacuum-min-free-fraction of the file. A compact
store is never rewritten, so the usual answer is three PRAGMAs and no I/O.
VACUUM takes SQLite's exclusive lock and rewrites the whole file (measured:
15 s and 2.4 GB -> 2.0 GB on a real store), so it runs OFF the open path —
vacuum-async! on a lowest-priority daemon thread after a settling delay,
never inside a transaction. Readers and writers in this process or another
one block for its duration and then continue, which the 30 s busy timeout and
the write-retry ladder in core absorb. A process that exits first, or a
db-close! that aborts the lease mid-rewrite, simply leaves the store due at
the next start: the rewrite is transactional, so an interrupted VACUUM rolls
back rather than damaging anything.
The vis.db.vacuum marker beside the store is BOTH the clock (its mtime is
the last successful vacuum) and the cross-process mutex (an exclusive
FileLock held for the rewrite). It is created only when a vacuum is
actually attempted and deleted again when one fails, so a store that never
needed reclaiming carries no marker and stays due.
Space reclamation for a file-backed SQLite store. SQLite never hands the file back on its own: `auto_vacuum` is off, so pages a delete or a `DROP COLUMN` frees go on the FREELIST and are only reused by later writes. That is the right default — reuse costs nothing — but a one-off bulk reclaim (retiring a column across a whole transcript) leaves hundreds of megabytes the file keeps until someone runs `VACUUM`. `maybe-vacuum!` is that someone, on the same fortnight window as `foundation.housekeeping` retention: at most once per `vacuum-interval-days` per store, and only when the freelist is worth the rewrite — at least `vacuum-min-free-bytes` AND `vacuum-min-free-fraction` of the file. A compact store is never rewritten, so the usual answer is three PRAGMAs and no I/O. VACUUM takes SQLite's exclusive lock and rewrites the whole file (measured: 15 s and 2.4 GB -> 2.0 GB on a real store), so it runs OFF the open path — `vacuum-async!` on a lowest-priority daemon thread after a settling delay, never inside a transaction. Readers and writers in this process or another one block for its duration and then continue, which the 30 s busy timeout and the write-retry ladder in `core` absorb. A process that exits first, or a `db-close!` that aborts the lease mid-rewrite, simply leaves the store due at the next start: the rewrite is transactional, so an interrupted VACUUM rolls back rather than damaging anything. The `vis.db.vacuum` marker beside the store is BOTH the clock (its mtime is the last successful vacuum) and the cross-process mutex (an exclusive `FileLock` held for the rewrite). It is created only when a vacuum is actually attempted and deleted again when one fails, so a store that never needed reclaiming carries no marker and stays due.
(free-space ds){:page-size :page-count :freelist-count :file-bytes :free-bytes} as SQLite
itself reports it — :file-bytes is the live file, :free-bytes what a
VACUUM would give back.
`{:page-size :page-count :freelist-count :file-bytes :free-bytes}` as SQLite
itself reports it — `:file-bytes` is the live file, `:free-bytes` what a
VACUUM would give back.(maybe-vacuum! ds db-file)(maybe-vacuum! ds
db-file
{:keys [now-ms interval-days min-free-bytes min-free-fraction]})Reclaim the freelist of the store at db-file when the fortnight has passed
AND the freelist is worth a rewrite. Returns a report — :is-vacuumed with a
:reason of :vacuumed, :recent (marker inside the window), :compact
(nothing worth reclaiming), :locked (another process is doing it) or
:failed — and never throws.
Options, all for tests: :now-ms, :interval-days, :min-free-bytes,
:min-free-fraction.
Reclaim the freelist of the store at `db-file` when the fortnight has passed AND the freelist is worth a rewrite. Returns a report — `:is-vacuumed` with a `:reason` of `:vacuumed`, `:recent` (marker inside the window), `:compact` (nothing worth reclaiming), `:locked` (another process is doing it) or `:failed` — and never throws. Options, all for tests: `:now-ms`, `:interval-days`, `:min-free-bytes`, `:min-free-fraction`.
(vacuum-async! ds db-file)(vacuum-async! ds db-file {:keys [delay-ms] :as opts})Fire-and-forget maybe-vacuum! for the store at db-file, on a
lowest-priority daemon thread that first sleeps :delay-ms so the rewrite is
not what a starting process waits on. Returns the thread. Called once per
store per process, from the pool that just opened it.
Fire-and-forget `maybe-vacuum!` for the store at `db-file`, on a lowest-priority daemon thread that first sleeps `:delay-ms` so the rewrite is not what a starting process waits on. Returns the thread. Called once per store per process, from the pool that just opened it.
Days between two reclaims of one store. The fortnight
foundation.housekeeping/default-retention-days already uses for every other
self-bounding artifact: long enough that no ordinary week ever pays the
rewrite, short enough that space freed by a migration comes back while the
operator still remembers the upgrade.
Days between two reclaims of one store. The fortnight `foundation.housekeeping/default-retention-days` already uses for every other self-bounding artifact: long enough that no ordinary week ever pays the rewrite, short enough that space freed by a migration comes back while the operator still remembers the upgrade.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |