S3 based konserve backend.
S3 based konserve backend.
When bound to an atom, S3 ops on THIS thread record into it. nil => off. Note: datahike's :self writer commits on its own thread, so a dynamic binding won't capture those PUTs — use the global accumulator below to measure across threads.
When bound to an atom, S3 ops on THIS thread record into it. nil => off. Note: datahike's :self writer commits on its own thread, so a dynamic binding won't capture those PUTs — use the global accumulator below to measure across threads.
(build-s3-client opts)Construct a fresh AWS S3Client from a connection config. Prefer s3-client,
which shares clients across stores.
Construct a fresh AWS S3Client from a connection config. Prefer `s3-client`, which shares clients across stores.
(common-client-config client
{:keys [region x-ray? access-key secret endpoint-override
path-style-access? expect-continue?]})(connect-store s3-spec & {:keys [opts]})Connect a konserve store backed by S3.
Connect a konserve store backed by S3.
(get-object-with-etag client bucket key)Get object and return map with :data and :etag, or nil if not found.
Get object and return map with :data and :etag, or nil if not found.
Holds nil or an accumulator atom. When set, S3 ops on ANY thread record into it. Use for measuring work that crosses threads (e.g. a datahike commit dispatched to the writer actor).
Holds nil or an accumulator atom. When set, S3 ops on ANY thread record into it. Use for measuring work that crosses threads (e.g. a datahike commit dispatched to the writer actor).
(io-stats-summary m)Reduce a raw io-stats atom value to {op {:n :total-ms :p50-ms :p99-ms}}.
Reduce a raw *io-stats* atom value to {op {:n :total-ms :p50-ms :p99-ms}}.
(list-objects client bucket)List ALL object keys in the bucket, following V2 continuation tokens. (The previous implementation issued a single ListObjects call and silently returned only the first 1000 keys.)
List ALL object keys in the bucket, following V2 continuation tokens. (The previous implementation issued a single ListObjects call and silently returned only the first 1000 keys.)
(list-stores s3-spec & {:keys [opts]})List all konserve stores in an S3 bucket by scanning per-store marker objects (<store-id>_.konserve-metadata). No central registry is kept, so concurrent store creation never contends on a shared object.
Args: s3-spec - Map with :bucket, :region, :access-key, :secret, etc. opts - (optional) Runtime options map. Defaults to {:sync? true}
Returns: Set of UUIDs representing store IDs in the bucket
List all konserve stores in an S3 bucket by scanning per-store marker
objects (<store-id>_.konserve-metadata). No central registry is kept, so
concurrent store creation never contends on a shared object.
Args:
s3-spec - Map with :bucket, :region, :access-key, :secret, etc.
opts - (optional) Runtime options map. Defaults to {:sync? true}
Returns:
Set of UUIDs representing store IDs in the bucket(put-object-conditional client bucket key bytes if-match-etag)Put object with conditional ETag check. Returns true on success, false on conflict. S3 returns HTTP 412 (Precondition Failed) when ifMatch ETag doesn't match.
Put object with conditional ETag check. Returns true on success, false on conflict. S3 returns HTTP 412 (Precondition Failed) when ifMatch ETag doesn't match.
(release _store env)Historically closed the per-store S3Client. S3Clients are now shared across
stores (see s3-client / client-cache), so per-store release is a no-op.
Call shutdown-clients! on process exit to close the shared clients.
Historically closed the per-store S3Client. S3Clients are now shared across stores (see `s3-client` / `client-cache`), so per-store release is a no-op. Call `shutdown-clients!` on process exit to close the shared clients.
(s3-client opts)Return a shared S3Client for the given connection config, constructing one
on first use and caching it per endpoint+credentials. Safe to call from many
threads and many stores — they all share one client per endpoint. Call
shutdown-clients! to close the cached clients (e.g. on process exit).
Return a shared S3Client for the given connection config, constructing one on first use and caching it per endpoint+credentials. Safe to call from many threads and many stores — they all share one client per endpoint. Call `shutdown-clients!` to close the cached clients (e.g. on process exit).
(set-global-io-stats! acc)Install acc (an atom) as the cross-thread IO accumulator, or nil to disable.
Install acc (an atom) as the cross-thread IO accumulator, or nil to disable.
(shutdown-clients!)Close every cached S3Client and clear the cache. Call once on process
shutdown. After this, s3-client constructs fresh clients again.
Close every cached S3Client and clear the cache. Call once on process shutdown. After this, `s3-client` constructs fresh clients again.
The SDK forces Expect: 100-continue onto PutObject via an internal
interceptor regardless of HTTP-client configuration (aws/aws-sdk-java-v2
#6537). The handshake costs a full extra round trip per PUT — and stalls
against providers that never send the interim 100 response (Cloudflare R2).
It only pays off for uploads large enough that resending a rejected body
hurts, which konserve values are not. Override-config interceptors run
after the SDK's own, so removing the header here wins.
The SDK forces `Expect: 100-continue` onto PutObject via an internal interceptor regardless of HTTP-client configuration (aws/aws-sdk-java-v2 #6537). The handshake costs a full extra round trip per PUT — and stalls against providers that never send the interim 100 response (Cloudflare R2). It only pays off for uploads large enough that resending a rejected body hurts, which konserve values are not. Override-config interceptors run after the SDK's own, so removing the header here wins.
(with-global-io-stats & body)Evaluate body with a fresh cross-thread accumulator installed, capturing S3 ops performed on any thread (e.g. datahike's writer actor). NOT reentrant / not concurrency-safe — use for isolated single-writer probes. Returns {:result :stats :raw} like with-io-stats.
Evaluate body with a fresh cross-thread accumulator installed, capturing S3
ops performed on any thread (e.g. datahike's writer actor). NOT reentrant /
not concurrency-safe — use for isolated single-writer probes. Returns
{:result :stats :raw} like with-io-stats.(with-io-stats & body)Evaluate body with a fresh io-stats atom bound on THIS thread. Returns {:result <body-value> :stats <io-stats-summary> :raw <atom-value>}. Same-thread only — see with-global-io-stats for cross-thread work.
Evaluate body with a fresh *io-stats* atom bound on THIS thread. Returns
{:result <body-value> :stats <io-stats-summary> :raw <atom-value>}.
Same-thread only — see with-global-io-stats for cross-thread work.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |