Konserve API Walkthrough

The big picture

• persist key-value pairs with schemaless data, typically serialized edn
• keys are created via hasch such that the keys themselves can be arbitrary edn
• the konserve.core api feels just like a clojure map except it has mutable persistence behind it
• the core functions can be used synchronously or asynchronously
• the details of the host system are abstracted away via protocols into host store implementations. sync vs async support is host dependent.
• konserve provides 4 built in stores
  • konserve.filestore
  • konserve.node-filestore
  • konserve.indexeddb
  • konserve.memory
• there are many others:
  • https://github.com/replikativ/konserve-s3
  • https://github.com/replikativ/konserve-jdbc
  • https://github.com/replikativ/konserve-redis
  • https://github.com/alekcz/konserve-fire
  • https://github.com/The-Literal-Company/konserve-cloud-storage
  • https://github.com/search?q=konserve&type=repositories&p=1

---

Connecting Stores

(require '[konserve.filestore :refer [connect-fs-store]])

(def store (<!! (connect-fs-store "/tmp/store")))



(require '[konserve.node-filestore :refer [connect-fs-store]])
;; node-js supports sync but no <!!
(def store (connect-fs-store "/tmp/store" :opts {:sync? true}))



;; in the browser
(require '[konserve.indexeddb :refer [connect-idb-store]])

(go
  ;; indexeddb is async only!
  (let [store (<! (connect-idb-store "idb-store"))]
    ...))

Stores also accept var-args. Support for each entry varies per implementation

• :opts => {:sync? <boolean>}

  • This is an env map passed around by most functions internally within konserve. The only entry you should typically need to concern yourself with is :sync? which is used to control whether functions return channels or values
  • an opts map is the last parameter accepted by konserve.core functions, but for creating stores, it must be identified by the keyword :opts

• :config => map

  • this map includes options for manipulating blobs in store specific ways. Very rarely should you ever need to alter the defaults

• :buffer-size => number

  • in clj this lets you control the chunk size used for writing blobs. the default is 1mb

• :default-serializer => keyword

  • the default serializer is :FressianSerializer, but you can override with a keyword identifying a different serializer implementation
  • (connect-store store-name :default-serializer :StringSerializer) => writes string edn
  • jvm also supports :CBORSerializer
  • you can provide your own serializer by giving a map of {:MySerializer PStoreSerializerImpl} to :serializers (..see next bullet) and then referencing it via :default-serializer :MySerializer

• :serializers => Map<keyword, PStoreSerializerImpl>

  • this is where you can provide your own serializer to reference via :default-serializer
  • konserve.serializers/fressian-serializer is a convenience function that accepts 2 maps: a map of read-handlers and a map of write-handlers and returns a fressian serializer supporting your custom types
    • in clj the handlers are reified org.fressian.ReadHandlers & org.fressian.WriteHandlers
      • see https://github.com/clojure/data.fressian/wiki/Creating-custom-handlers
    • in cljs handlers are just functions
      • see https://github.com/pkpkpk/fress

• :encryptor => {:type :aes :key "s3cr3t"}

  • currently only supports :aes in default stores

• :compressor => {:type :lz4}

  • currently LZ4 compression is only supported on the jvm

Incognito & Records

Konserve intercepts records and writes them as incognito tagged literals such that the details of serialization formats are abstracted away and allowing easier interop between different formats. The :read-handlers and :write-handlers varg args are explicitly meant for working with incognito's tagged literals.

• :read-handlers expects an atom wrapping {'symbol.for.MyRecord map->MyRecord} for recovering records from incognito tagged literals
• :write-handlers expects an atom wrapping {'symbol.for.MyRecord (fn [record] ..)} for writing records as incognito tagged literals
• the symbols used in these maps are not safe for clojurescript so you should avoid using them

---

Working with data

Once you have a store you can access it using konserve.core functions. By default functions are asynchronous and return channels yielding values or errors. You can override this by passing an opts map with :sync? true

(require '[konserve.core :as k])

(k/exists? store :false) ;=> channel<false>

(<!! (k/assoc store :fruits {:parcha nil :mango nil :banana nil}))

(k/exists? store :fruits {:sync? true}) ;=> true

You can get get-in update update-in and dissoc just like a clojure map.

(k/assoc-in store [:fruits :parcha] {:color "yellow" :taste "sour" :quantity 0})

(k/update-in store [:fruits :parcha :quantity] inc)

(k/get-in store [:fruits :parcha :quantity]) ;=> channel<1>

(k/dissoc store :fruits)

In the fruits example a simple keyword is the store key, but keys themselves can be arbitrary edn:

(defn memoize-to-store-sync [f]
  (fn [& args]
    (if-let [result (<!! (k/get store args))]
      result
      (let [result (apply f args)]
        (<!! (k/assoc store args result))
        result))))

(def memoized-fn (memoize-to-store-sync expensive-fn))

(memoized-fn {:any/such #{"set"}}, [0x6F \f], 'haschable.argu/ments) ;=> channel<result>

Working with binary data

(k/bassoc store :blob blob)

(k/bget store :blob
              (fn locked-cb [{is :input-stream}]
                (go (input-stream->byte-buffer is)))) ;=> ch<bytebuffer>

With bassoc binary data is written as-is without passing through serialization/encryption/compression.

bget is probably the trickiest function in konserve. It accepts a callback function that is passed a map of {:input-stream <host-input-stream>}. While locked-cb is running, konserve locks & holds onto underyling resources (ie file descriptors) until the function exits. You can choose to read from the input stream however you like, but rather than running side-effects within the callback, you should instead return your desired value else the lock will remain held.

• when called async, the locked-cb should return a channel yielding the desired value that will be read from and yielded by bget's channel
• in both clojurescript stores, synchronous input streams are not possible.
• On nodejs you can call bget synchronously but the locked-cb will be called with {:blob js/Buffer}
• In the browser with indexedDB, the async only bget calls its locked-cb with {:input-stream <readable-webstream> :offset <number>} where offset indicates the amount of bytes to drop before reaching the desired blob start.
  • konserve.indexeddb/read-web-stream can serve as a locked-cb that will yield a Uint8Array.

Metadata

Konserve does some bookkeeping for values by storing them with metadata

(k/get-meta store :key {:sync? true})
;=>
;  {:key :key
;   :type <binary|edn>
;   :last-write <inst>}

The append log

Konserve provides an append log for writing values quickly. These entries are a special case managed by konserve, where the sequence is stored as a linked list of blobs where each blob is a cons cell of the list. You can name the log with any key, but that key should only be written to or read from using the append, log, and reduce-log functions.

(dotimes [n 6]
  (<!! (k/append store :log n)))

(k/get store :log) ;=> channel<(0 1 2 3 4 5)>

(k/reduce-log store :log
              (fn [acc n]
                (if (even? n)
                  (conj acc n)
                  acc))
              []) ;=> channel<[0 2 4]>

konserve.gc

konserve.gc/sweep! lets you prune the store based on a whitelist set of keys to keep and a timestamp cutoff before which un whitelisted entries should be deleted

konserve.cache

konserve.cache/ensure-cache wraps a store with a lru-cache to avoid hitting external memory for frequently accessed keys