taoensso.carmine.tundra — zaden/carmine 2.18.1.2

Semi-automatic archival datastore layer for Carmine. Use multiple Redis instances (recommended) or Redis databases for local key namespacing.

Redis keys:

• carmine:tundra:evictable - set, ks for which ensure-ks fail should throw

<urlsafe-str clj

(<urlsafe-str s)

>urlsafe-str clj

(>urlsafe-str s)

catcht clj macro

(catcht & body)

dirty clj

(dirty tstore & ks)

Queues given keys for freezing to datastore. Throws an exception if any keys don't exist. Acts as a Redis command: call within a wcar context.

If TundraStore has a :redis-ttl-ms option, MARKS GIVEN KEYS FOR EXPIRY!! ** Worker MUST be running AND FUNCTIONING CORRECTLY or DATA WILL BE LOST! **

ensure-ks clj

(ensure-ks tstore & ks)

BLOCKS to ensure given keys are available in Redis, fetching them from datastore as necessary. Throws an exception if any previously evicted keys couldn't be made available. Acts as a Redis command: call within a wcar context.

extend-exists-missing-ks clj

(extend-exists-missing-ks ttl-ms ks & [only-evictable?])

fetch-keys-delayed clj

Used to prevent multiple threads from rushing the datastore to get the same keys, unnecessarily duplicating work.

IDataStore clj protocol

Extension point for additional datastores.

fetch-keys clj

(fetch-keys dstore ks)

(fetch-keys dstore ["key" ...]) => [<#{<frozen-val> <ex>}> ...]

put-key clj

(put-key dstore k v)

(put-key dstore "key" <frozen-val>) => <#{true <ex>}>

IFreezer clj protocol

Extension point for compressors, encryptors, etc.

freeze clj

(freeze freezer x)

Returns datastore-ready key val. (comp put-key freeze): dump-ba -> datastore val.

thaw clj

(thaw freezer x)

Returns Redis-ready key val. (comp thaw fetch-key): datastore val -> dump-ba.

ITundraStore clj protocol

dirty* clj

(dirty* tstore ks)

ensure-ks* clj

(ensure-ks* tstore ks)

worker clj

(worker tstore conn-opts wopts)

Alpha - subject to change. Returns a threaded message queue worker to routinely freeze Redis keys marked as dirty to datastore and mark successfully frozen keys as clean. Logs any errors. THESE ERRORS ARE IMPORTANT: an email or other appropriate notification mechanism is HIGHLY RECOMMENDED. If a worker shuts down and your keys are set to expire YOU WILL IRREVOCABLY LOSE DATA.

Options: :nattempts - Number of times worker will attempt to freeze a key to datastore before failing permanently. >=1. :retry-backoff-ms - Amount of time (msecs) to backoff before retrying a failed key freeze. >=0. Can be a (fn [attempt]) -> ms.

:montior, :eoq-backoff-ms, :nthreads, :throttle-ms, :auto-start

• Standard taoensso.carmine.message-queue/worker opts.

nippy-freezer clj

Default Nippy Freezer.

tundra-store clj

(tundra-store datastore
              &
              [{:keys [tqname freezer redis-ttl-ms]
                :or {tqname :default freezer nippy-freezer}}])

Alpha - subject to change. Returns a TundraStore with options: datastore - Storage for frozen key data. Default datastores: taoensso.carmine.tundra.faraday/faraday-datastore taoensso.carmine.tundra.s3/s3-datastore. :tqname - Optional. Worker message queue name. :freezer - Optional. Preps key data to/from datastore. May provide services like compression and encryption, etc. Defaults to Nippy with default options (Snappy compression and no encryption). :redis-ttl-ms - Optional! Time after which frozen, inactive keys will be EVICTED FROM REDIS (DELETED!). Minimum 10 hours. ONLY use this if you have CONFIRMED that your worker is successfully freezing the necessary keys to your datastore. Otherwise YOU WILL IRREVOCABLY LOSE DATA.

See ensure-ks, dirty, worker for TundraStore API.