Space reclamation for a file-backed SQLite store.
SQLite never hands the file back on its own: auto_vacuum is off, so pages a
delete or a DROP COLUMN frees go on the FREELIST and are only reused by
later writes. That is the right default — reuse costs nothing — but a one-off
bulk reclaim (retiring a column across a whole transcript) leaves hundreds of
megabytes the file keeps until someone runs VACUUM.
maybe-vacuum! is that someone, on the same fortnight window as
foundation.housekeeping retention: at most once per vacuum-interval-days
per store, and only when the freelist is worth the rewrite — at least
vacuum-min-free-bytes AND vacuum-min-free-fraction of the file. A compact
store is never rewritten, so the usual answer is three PRAGMAs and no I/O.
VACUUM takes SQLite's exclusive lock and rewrites the whole file (measured:
15 s and 2.4 GB -> 2.0 GB on a real store), so it runs OFF the open path —
vacuum-async! gives it to internal.janitor, the one maintenance worker of
a process, after a settling delay, never inside a transaction. Readers and
writers in this process or another one block for its duration and then
continue, which the 30 s busy timeout and the write-retry ladder in core
absorb. A process that exits first, or a db-close! that aborts the lease
mid-rewrite, simply leaves the store due at the next start: the rewrite is
transactional, so an interrupted VACUUM rolls back rather than damaging
anything.
The vis.db.vacuum marker beside the store is BOTH the clock (its mtime is
the last successful vacuum) and the cross-process mutex (an exclusive
FileLock held for the rewrite). It is created only when a vacuum is
actually attempted and deleted again when one fails, so a store that never
needed reclaiming carries no marker and stays due.
Space reclamation for a file-backed SQLite store. SQLite never hands the file back on its own: `auto_vacuum` is off, so pages a delete or a `DROP COLUMN` frees go on the FREELIST and are only reused by later writes. That is the right default — reuse costs nothing — but a one-off bulk reclaim (retiring a column across a whole transcript) leaves hundreds of megabytes the file keeps until someone runs `VACUUM`. `maybe-vacuum!` is that someone, on the same fortnight window as `foundation.housekeeping` retention: at most once per `vacuum-interval-days` per store, and only when the freelist is worth the rewrite — at least `vacuum-min-free-bytes` AND `vacuum-min-free-fraction` of the file. A compact store is never rewritten, so the usual answer is three PRAGMAs and no I/O. VACUUM takes SQLite's exclusive lock and rewrites the whole file (measured: 15 s and 2.4 GB -> 2.0 GB on a real store), so it runs OFF the open path — `vacuum-async!` gives it to `internal.janitor`, the one maintenance worker of a process, after a settling delay, never inside a transaction. Readers and writers in this process or another one block for its duration and then continue, which the 30 s busy timeout and the write-retry ladder in `core` absorb. A process that exits first, or a `db-close!` that aborts the lease mid-rewrite, simply leaves the store due at the next start: the rewrite is transactional, so an interrupted VACUUM rolls back rather than damaging anything. The `vis.db.vacuum` marker beside the store is BOTH the clock (its mtime is the last successful vacuum) and the cross-process mutex (an exclusive `FileLock` held for the rewrite). It is created only when a vacuum is actually attempted and deleted again when one fails, so a store that never needed reclaiming carries no marker and stays due.
(free-space ds){:page-size :page-count :freelist-count :file-bytes :free-bytes} as SQLite
itself reports it — :file-bytes is the live file, :free-bytes what a
VACUUM would give back.
`{:page-size :page-count :freelist-count :file-bytes :free-bytes}` as SQLite
itself reports it — `:file-bytes` is the live file, `:free-bytes` what a
VACUUM would give back.(maybe-vacuum! ds db-file)(maybe-vacuum! ds
db-file
{:keys [now-ms interval-days min-free-bytes min-free-fraction]})Reclaim the freelist of the store at db-file when the fortnight has passed
AND the freelist is worth a rewrite. Returns a report — :is-vacuumed with a
:reason of :vacuumed, :recent (marker inside the window), :compact
(nothing worth reclaiming), :locked (another process is doing it) or
:failed — and never throws.
Options, all for tests: :now-ms, :interval-days, :min-free-bytes,
:min-free-fraction.
Reclaim the freelist of the store at `db-file` when the fortnight has passed AND the freelist is worth a rewrite. Returns a report — `:is-vacuumed` with a `:reason` of `:vacuumed`, `:recent` (marker inside the window), `:compact` (nothing worth reclaiming), `:locked` (another process is doing it) or `:failed` — and never throws. Options, all for tests: `:now-ms`, `:interval-days`, `:min-free-bytes`, `:min-free-fraction`.
(vacuum-async! ds db-file)(vacuum-async! ds db-file {:keys [delay-ms before open?] :as opts})Give maybe-vacuum! for the store at db-file to the janitor, which runs it
after :delay-ms, so the rewrite is not what a starting process waits on. A
:before function runs first in the same task, so a data rewrite frees its
pages before the reclaim checks them. :open? answers whether the store is
still open: a store that closed in the delay is due at its next open. Called
once per store per process, from the pool that just opened it. A second call
for one store replaces a task that still waits.
The janitor writes the report of a vacuum that ran or that another process holds. A store with nothing to reclaim leaves no line, and a failed vacuum is a warning.
Give `maybe-vacuum!` for the store at `db-file` to the janitor, which runs it after `:delay-ms`, so the rewrite is not what a starting process waits on. A `:before` function runs first in the same task, so a data rewrite frees its pages before the reclaim checks them. `:open?` answers whether the store is still open: a store that closed in the delay is due at its next open. Called once per store per process, from the pool that just opened it. A second call for one store replaces a task that still waits. The janitor writes the report of a vacuum that ran or that another process holds. A store with nothing to reclaim leaves no line, and a failed vacuum is a warning.
Days between two reclaims of one store. The fortnight
foundation.housekeeping/default-retention-days already uses for every other
self-bounding artifact: long enough that no ordinary week ever pays the
rewrite, short enough that space freed by a migration comes back while the
operator still remembers the upgrade.
Days between two reclaims of one store. The fortnight `foundation.housekeeping/default-retention-days` already uses for every other self-bounding artifact: long enough that no ordinary week ever pays the rewrite, short enough that space freed by a migration comes back while the operator still remembers the upgrade.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |