Codax is an idiomatic transactional embedded database for clojure. A codax database functions as a single (potentially enormous) clojure map, with a similar access & modification api.
Version 1.2.0 adds seeking features. See CHANGELOG for more details.
Even simple programs frequently benefit from saving data to disk. Unfortunately, there is generally a large semantic leap once you go from values in memory to values on disk. Codax aims to close that semantic gap. While it won't win any speed contests, it is designed to be performant enough for applications serving thousands of users. Most importantly, it is designed to make data persistance as low friction as possible. It is also designed to be effortless to get started with. There are no external libraries to install, and the underlying B+ tree is written directly in clojure.
Fundamentally, I wrote this library for myself in an ongoing effort to make my own life simpler and my own programming more fun. I wanted to share it with the community in the hopes that others may find it does the same for them.
Codax provides the following guarantees:
I have successfully used this library in production environments. That said, there are probably a few rough edges that could use smoothing.
Database Functions
open-database!
- Opens or creates a database, or returns an existing database connection if it's already openclose-database!
- Safely closes an open databaseclose-all-databases!
- Safely closes all open databasesis-open?
- Checks if a database is opendestroy-database!
- Deletes a database and all its data irretrievably (intended for use in tests).Transaction Macros
These take a database argument and a transaction-symbol and bind the symbol to a newly created transaction. Transactions are isolated from each other. Read-transactions evaluate to the value of their body, but a (successful) write-transaction evaluates to nil.
with-read-transaction
- creates a read transactionwith-write-transaction
- creates a write transaction (body must evaluate to a transaction or an exception will be thrown)In-Transaction Functions
These are all similar to the clojure.core map *-in
(e.g. assoc-in
) with the following exceptions:
These must be called within a with-write-transaction
or a with-read-transaction
expression. Changes will only be persistent if with-write-transaction
is used.
get-at
assoc-at
update-at
merge-at
dissoc-at
Shortcut Functions
These are the same as the transactional-functions except that their first argument is a database instead of a transaction. These are convenience functions which automatically create and execute transactions. The write variants will also return the result of the modification.
get-at!
assoc-at!
update-at!
merge-at!
dissoc-at!
Seek Functions added in 1.2.0
These allow you to get ordered subsets of data from the database "map" in the form of ordered key-value pairs. Each accepts optional :limit
and :reverse
keyword parameters. They follow the same naming conventions of the other functions (plain variants expect a tx
argument and !
variants expect a db
argument).
seek-at
& seek-at!
- get ordered key-value pairs from the map at the provided path
seek-from
& seek-from!
- get ordered key-value pairs from the map at the provided path for keys >= start-val
seek-to
& seek-to!
- get ordered key-value pairs from the map at the provided path for keys <= end-val
seek-range
& seek-range!
- get ordered key-value pairs from the map at the provided path for keys >= start-val
and <= end-val
seek-prefix
& seek-prefix!
- get ordered key-value pairs from the map at the provided path for string or keyword keys which begin with val-prefix
seek-prefix-range
& seek-prefix-range!
- get ordered key-value pairs from the map at the provided path for string or keyword keys which begin with a value between (inclusive) start-prefix
& end-prefix
See Seek Examples
A path
is a vector of keys similar to the [k & ks]
used in function like assoc-in
with a few exceptions:
(get-at db [])
returns the full database) but you cannot modify it (e.g. (assoc-at [] :foo)
throws an error)If you are interested in contributing support for additional types, please review doc/types.md.
Immutability
Transactions are immutable. Each transformation (e.g. assoc-at
, update-at
) returns a new transaction, it does not modify the transaction. Essentially you should treat them as you would a standard clojure map, one that you interact with using the *-at
functions.
Example:
(c/with-write-transaction [db tx-original]
(let [tx-a (c/assoc-at tx-original [:letter] "a")
tx-b (c/assoc-at tx-original [:letter] "b")]
tx-a))
(c/get-at! db [:letter]) ; "a"
See the FAQ for examples of potential pitfalls.
Visibility
Changes in a transaction are only visible to subsequent transformations on that transaction. They are not visible anywhere else until committed (by being the final result in the body of a with-write-transaction
expression). The changes are also not visible in any read transaction opened before the write transaction is committed.
Example:
(c/with-write-transaction [db tx]
(-> tx
(c/assoc-at [:number] 1000)
(c/update-at [:number] inc)))
(c/get-at! db [:number]) ; 1001
Exceptions
If an Exception is thrown within a with-write-transaction
expression, the transaction is aborted and no changes are persisted.
Locking
Write transactions block other write transactions (though they do not block read transactions). It is best to avoid doing any computationally complex or IO heavy tasks (such as fetching remote data) inside a with-write-transaction
block. See Performance for more details.
(require [codax.core :as c])
(def db (c/open-database! "data/demo-database")) ;
(c/assoc-at! db [:assets :people] {0 {:name "Alice"
:occupation "Programmer"
:age 42}
1 {:name "Bob"
:occupation "Writer"
:age 27}}) ; {0 {:age 42, :name "Alice", ...}, 1 {:age 27, :name "Bob", ...}}
(c/get-at! db [:assets :people 0]) ; {:name "Alice" :occupation "Programmer" :age 42}
(c/update-at! db [:assets :people 1 :age] inc) ; 28
(c/merge-at! db [:assets] {:tools {"hammer" true
"keyboard" true}}) ; {:people {...} :tools {"hammer" true, "keyboard" true}}
(c/get-at! db [:assets])
;; {:people {0 {:name "Alice"
;; :occupation "Programmer"
;; :age 42}
;; 1 {:name "Bob"
;; :occupation "Writer"
;; :age 28}}
;; :tools {"hammer" true
;; "keyboard" true}}
(c/close-database! db)
(def db (c/open-database! "data/demo-database"))
;;;; init
(c/with-write-transaction [db tx]
(c/assoc-at tx [:counters] {:id 0 :users 0}))
;;;; user fns
(defn add-user
"create a user and assign them an id"
[username]
(c/with-write-transaction [db tx]
(when (c/get-at tx [:usernames username] )
(throw (Exception. "username already exists")))
(let [user-id (c/get-at tx [:counters :id])
user {:id user-id
:username username
:timestamp (System/currentTimeMillis)}]
(-> tx
(c/assoc-at [:users user-id] user)
(c/assoc-at [:usernames username] user-id)
(c/update-at [:counters :id] inc)
(c/update-at [:counters :users] inc)))))
(defn get-user
"fetch a user by their username"
[username]
(c/with-read-transaction [db tx]
(when-let [user-id (c/get-at tx [:usernames username])]
(c/get-at tx [:users user-id]))))
(defn rename-user
"change a username"
[username new-username]
(c/with-write-transaction [db tx]
(when (c/get-at tx [:usernames new-username] )
(throw (Exception. "username already exists")))
(when-let [user-id (c/get-at tx [:usernames username])]
(-> tx
(c/dissoc-at [:usernames username])
(c/assoc-at [:usernames new-username] user-id)
(c/assoc-at [:users user-id :username] new-username)))))
(defn remove-user
"remove a user"
[username]
(c/with-write-transaction [db tx]
(when-let [user-id (c/get-at tx [:usernames username])]
(-> tx
(c/dissoc-at [:username username])
(c/dissoc-at [:users user-id])
(c/update-at [:counters :users] dec)))))
;;;;; edit users
(c/get-at! db) ; {:counters {:id 0, :users 0}}
(add-user "charlie") ; nil
(c/get-at! db)
;; {:counters {:id 1, :users 1},
;; :usernames {"charlie" 0},
;; :users {0 {:id 0, :timestamp 1484529469567, :username "charlie"}}}
(add-user "diane") ; nil
(c/get-at! db)
;; {:counters {:id 2, :users 2},
;; :usernames {"charlie" 0, "diane" 1},
;; :users
;; {0 {:id 0, :timestamp 1484529469567, :username "charlie"},
;; 1 {:id 1, :timestamp 1484529603444, :username "diane"}}}
(rename-user "charlie" "chuck") ; nil
(c/get-at! db)
;; {:counters {:id 2, :users 2},
;; :usernames {"chuck" 0, "diane" 1},
;; :users
;; {0 {:id 0, :timestamp 1484529469567, :username "chuck"},
;; 1 {:id 1, :timestamp 1484529603444, :username "diane"}}}
(remove-user "diane") ; nil
(c/get-at! db)
;; {:counters {:id 2, :users 1},
;; :usernames {"chuck" 0, "diane" 1},
;; :users {0 {:id 0, :timestamp 1484529469567, :username "chuck"}}}
(c/close-database! db)
Directory Example
(def db (c/open-database! "data/example-database"))
(c/assoc-at! db [:directory]
{"Alice" {:ext 247, :dept "qa"}
"Barbara" {:ext 228, :dept "qa"}
"Damian" {:ext 476, :dept "hr"}
"Adam" {:ext 357, :dept "hr"}
"Frank" {:ext 113, :dept "hr"}
"Bill" {:ext 234, :dept "sales"}
"Evelyn" {:ext 337, :dept "dev"}
"Chuck" {:ext 482, :dept "sales"}
"Emily" {:ext 435, :dept "dev"}
"Diane" {:ext 245, :dept "dev"}
"Chelsea" {:ext 345, :dept "qa"}
"Bob" {:ext 326, :dept "sales"}})
;; - seek-at -
(c/seek-at! db [:directory])
;;[["Adam" {:dept "hr", :ext 357}]
;; ["Alice" {:dept "qa", :ext 247}]
;; ["Barbara" {:dept "qa", :ext 228}]
;; ["Bill" {:dept "sales", :ext 234}]
;; ["Bob" {:dept "sales", :ext 326}]
;; ["Chelsea" {:dept "qa", :ext 345}]
;; ["Chuck" {:dept "sales", :ext 482}]
;; ["Damian" {:dept "hr", :ext 476}]
;; ["Diane" {:dept "dev", :ext 245}]
;; ["Emily" {:dept "dev", :ext 435}]
;; ["Evelyn" {:dept "dev", :ext 337}]
;; ["Frank" {:dept "hr", :ext 113}]]
(c/seek-at! db [:directory] :limit 3)
;;[["Adam" {:dept "hr", :ext 357}]
;; ["Alice" {:dept "qa", :ext 247}]
;; ["Barbara" {:dept "qa", :ext 228}]]
(c/seek-at! db [:directory] :limit 3 :reverse true)
;;[["Frank" {:ext 113, :dept "hr"}]
;; ["Evelyn" {:ext 337, :dept "dev"}]
;; ["Emily" {:ext 435, :dept "dev"}]]
;; - seek-prefix -
(c/seek-prefix! db [:directory] "B")
;;[["Barbara" {:dept "qa", :ext 228}]
;; ["Bill" {:dept "sales", :ext 234}]
;; ["Bob" {:dept "sales", :ext 326}]]
;; - seek-prefix-range -
(c/seek-prefix-range! db [:directory] "B" "D")
;;[["Barbara" {:dept "qa", :ext 228}]
;; ["Bill" {:dept "sales", :ext 234}]
;; ["Bob" {:dept "sales", :ext 326}]
;; ["Chelsea" {:dept "qa", :ext 345}]
;; ["Chuck" {:dept "sales", :ext 482}]
;; ["Damian" {:dept "hr", :ext 476}]
;; ["Diane" {:dept "dev", :ext 245}]]
(c/close-database! db)
Messaging Example
(def db (c/open-database! "data/example-database")) ;
(defn to-instant [s]
(java.time.Instant/ofEpochMilli (java.util.Date/parse s)))
(defn post-message!
([user body]
(post-message! (java.time.Instant/now) user body))
([inst user body]
(c/assoc-at! db [:messages inst] {:user user
:body body})))
(defn process-messages
[messages]
(map (fn [[inst m]] (assoc m :time (str inst))) messages))
(defn get-messages-before [ts]
(process-messages
(c/seek-to! db [:messages] ts)))
(defn get-messages-after [ts]
(process-messages
(c/seek-from! db [:messages] ts)))
(defn get-messages-between [start-ts end-ts]
(process-messages
(c/seek-range! db [:messages] start-ts end-ts)))
(defn get-recent-messages [n]
(-> (c/seek-at! db [:messages] :limit n :reverse true)
process-messages ;;
reverse)); we reverse the result because we want the messages to be in chronological order
; but we needed to use the :reverse seek parameter to prevent collecting all
; of the messages from the beginning of time (there could be many thousands!)
(defn simulate-message! [date-time-string user body]
(post-message! (to-instant date-time-string) user body))
(simulate-message! "June 6, 2020 11:01" "Bobby" "Hello")
(simulate-message! "June 6, 2020 11:02" "Alice" "Welcome, Bobby")
(simulate-message! "June 6, 2020 11:03" "Bobby" "I was wondering how codax seeking works?")
(simulate-message! "June 6, 2020 11:07" "Alice" "Please be more specific, have you read the docs/examples?")
(simulate-message! "June 6, 2020 11:08" "Bobby" "Oh, I guess I should do that.")
(simulate-message! "June 7, 2020 14:30" "Chuck" "Anybody here?")
(simulate-message! "June 7, 2020 14:35" "Chuck" "Guess not...")
(simulate-message! "June 8, 2020 16:50" "Bobby" "Okay, so I read the docs. What is the :reverse param for?")
(simulate-message! "June 8, 2020 16:55" "Alice" "Basically, it seeks from the end and works backwards")
(simulate-message! "June 8, 2020 16:56" "Bobby" "Why would I do that?")
(simulate-message! "June 8, 2020 16:57" "Alice" "Well, generally it is used to grab just the end of a long dataset.")
(get-recent-messages 3)
;;({:user "Alice" :time "2020-06-08T22:55:00Z" :body "Basically, it seeks from the end and works backwards"}
;; {:user "Bobby" :time "2020-06-08T22:56:00Z" :body "Why would I do that?"}
;; {:user "Alice" :time "2020-06-08T22:57:00Z" :body "Well, generally it is used to grab just the end of a long dataset." })
(get-messages-after (to-instant "June 7, 2020 14:32"))
;;({:user "Chuck" :time "2020-06-07T20:35:00Z" :body "Guess not..."}
;; {:user "Bobby" :time "2020-06-08T22:50:00Z" :body "Okay, so I read the docs. What is the :reverse param for?"}
;; {:user "Alice" :time "2020-06-08T22:55:00Z" :body "Basically, it seeks from the end and works backwards"}
;; {:user "Bobby" :time "2020-06-08T22:56:00Z" :body "Why would I do that?"}
;; {:user "Alice" :time "2020-06-08T22:57:00Z" :body "Well, generally it is used to grab just the end of a long dataset." })
(get-messages-before (to-instant "June 6, 2020 11:05"))
;;({:user "Bobby" :time "2020-06-06T17:01:00Z" :body "Hello"}
;; {:user "Alice" :time "2020-06-06T17:02:00Z" :body "Welcome, Bobby"}
;; {:user "Bobby" :time "2020-06-06T17:03:00Z" :body "I was wondering how codax seeking works?"})
(get-messages-between (to-instant "June 7, 2020")
(to-instant "June 7, 2020 23:59"))
;;({:user "Chuck" :time "2020-06-07T20:30:00Z" :body "Anybody here?"}
;; {:user "Chuck" :time "2020-06-07T20:35:00Z" :body "Guess not..."})
(c/close-database! db)
Why aren't all my changes being saved?
Because transactions are immutable, if an updated transaction is discarded, the transformations it contains will not be committed.
Incorrect:
(c/with-write-transaction [db tx]
(c/assoc-at tx [:users 1] "Alice") ; this write is "lost"
(c/assoc-at tx [:users 2] "Bob"))
(c/get-at! db [:users]) ; {2 "Bob"}
Correct:
(c/with-write-transaction [db tx]
(-> tx ; thread the transaction through multiple transformations
(c/assoc-at [:users 1] "Alice")
(c/assoc-at [:users 2] "Bob")))
(c/get-at! db [:users]) ; {1 "Alice" 2 "Bob"}
Why am I getting a NullPointerException in my Write Transaction?
A common cause is that the body of the with-write-transaction
form is not evaluating to (returning) a transaction.
Incorrect:
(defn init-counter! []
(c/with-write-transaction [db tx]
(when-not (c/get-at tx [:counter])
(c/assoc-at tx [:counter] 0))))
(init-counter!) ; nil (it works the first time)
(init-counter!) ; java.lang.NullPointerException (the body evaluates to nil)
Correct:
(defn init-counter! []
(c/with-write-transaction [db tx]
(if-not (c/get-at tx [:counter])
(c/assoc-at tx [:counter] 0)
tx))) ;; if the counter is already initialized, return the unmodified transaction
(init-counter!) ; nil
(init-counter!) ; nil
Codax is geared towards read-heavy workloads.
Jan 14, 2017
The following figures are for a database populated with 16,000,000 (map-leaf) values running on a Digital Ocean 2-core 2GB RAM instance. The write transactions have an average "path" length of 6 and an average 7 leaf values.
These values come from running the codax.bench.performance/run-benchmark
benchmarking function without arguments 3 times consecutively.
...
Insights, suggestions, and PRs are very welcome.
Copyright © 2017 David Scarpetti
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.
Can you improve this documentation?Edit on GitHub
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close