In Eva 101, we delved into the fundamentals of Eva. We showed how schema was defined and transacted into the database. We showed how transactions are also used to add data to the system. Finally we went on a whirlwind tour of the query faculties provided by Eva.
Today we will expose the basics of database functions and exceptions, look at various historical views of the database, and explore some API's in Eva that allow you to manipulate your data differently.
First we will start with our schema, modeling that of account holders at a bank, and add a few entities to go along with it.
Schema and Entities
(def schema
[{:db/id (eva/tempid :db.part/db)
:db/ident :account/name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "An account's name"
:db.install/_attribute :db.part/db}
{:db/id (eva/tempid :db.part/db)
:db/ident :account/balance
:db/cardinality :db.cardinality/one
:db/valueType :db.type/long
:db/doc "The accounts balance"
:db.install/_attribute :db.part/db}
])
(def records
[{:db/id (eva/tempid :db.part/user -1) :account/name "Jeff Bridges"}
{:db/id (eva/tempid :db.part/user -1) :account/balance 100}
{:db/id (eva/tempid :db.part/user -2) :account/name "Jimmy Fallon"}
{:db/id (eva/tempid :db.part/user -2) :account/balance 1000}
{:db/id (eva/tempid :db.part/user -3) :account/name "Michael Jackson"}
{:db/id (eva/tempid :db.part/user -3) :account/balance 10000}])
(def conn (eva/connect {:local true}))
@(eva/transact conn schema)
@(eva/transact conn records)
Pretty simple, we have a couple accounts with names and a starting balance.
(def get-entity-id '[:find ?e . :in $ ?name :where [?e :account/name ?name]])
get-entity-id performs a query which takes a name and returns the entity id for the associated account. Let's add 100 dollars to the account of Jeff Bridges.
(def db (eva/db conn))
@(eva/transact conn [[:db.fn/cas
(eva/q get-entity-id db "Jeff Bridges")
:account/balance
100
200]])
Wait, :db.fn/cas, what is that? Compare-and-swap is a built-in database function. It is used to update the value of a single datom, taking as arguments an entity id, attribute, an old value, and a new value. A :db.fn/cas operation will succeed only if the old value you provide matches that which is found in the database at the time of the transaction.
This introduces somewhat of a problem, as we would ideally like to be able to update the balance without worrying about whether or not someone came in and modified the balance before us. Enter the transaction function.
Transaction Functions
Transaction functions are a subset of database functions which run inside a transaction. They must accept a db value as their first argument, and return a valid list of transaction data. In this case we are going to define and install a function that will run as part of our transaction to ensure atomicity when incrementing the balance of an account:
(def inc-balance
[{:db/id (eva/tempid :db.part/user -1)
:db/ident :inc-balance
:db/doc "Data function that increments value of attribute by an amount."
:db/fn (eva/function {:lang "clojure"
:params '[db e amount]
:code '[[:db/add e :account/balance
(-> (d/entity db e) :account/balance (+ amount))]]
})}
])
@(eva/transact conn inc-balance)
Firstly, notice the similarities between this function and the functions we've seen up to this point. We re-use a number of :db keywords such as id, identity, and docstring. The new one here is :db/fn, which is just a map used to describe the function.
Enclosed in :db/fn is :lang, which defines the language we wish to write our function in. Currently we only support "clojure" but we expect to support "java" sometime in the future. The first argument to the :params keyword must be a db value, followed by any other parameters (up to 20) that you want to pass to the function. In this case we are passing an entity id e, and the amount we wish to add to the account's balance.
Let's go through the logic inside the :code block. The first part we've seen before, and is simply the signature for :db/add except that the value is derived from (-> (d/entity db e) :account/balance (+ amount)).
-> is a thread-first macro we use to pass the result of a form as the first argument to the next form. It allows us to read code in an imperative style. So first, we fetch the entity e from the database, then read off :account/balance from that entity and then finally add to the read balance the amount.
Using -> also makes the code easier to read than the following (which is the same statement expressed without ->):
(+ (:account/balance (d/entity db e)) amount)
One important thing to note is that inside the code block of a transaction function, we alias d as the namespace when accessing Eva API functions.
Entity API
Another thing that we haven't seen before is eva/entity. This function takes a db value and an entity-id and gives you back the entity. We can use it like:
(def ent (eva/entity db (eva/q get-entity-id db "Michael Jackson")))
Keep in mind, the entity is lazy and the values of its attributes are not obtained from the db until we try to access them explicitly. If we just want to de-serialize a single attribute, we can do:
(ent :account/name)
Or, if we want to de-serialize the entire entity, we can use touch.
(eva/touch ent)
Now that all the pieces are in place, let's update the balance for the account using the transaction function :inc-balance we created.
@(eva/transact conn [[:inc-balance
(eva/q get-entity-id db "Jeff Bridges")
30]])
There you have it, we've successfully updated the balance of the account by 30. We simply call the :inc-balance keyword as part of the transaction, supplying it with our entity id and the amount by which we want the account balance incremented. The db value is passed implicitly to the transaction function, so it does not need to be explicitly specified.
We can take this a step further by creating another transaction function so that we can transfer between existing accounts.
(def transfer
[{:db/id (eva/tempid :db.part/user -1)
:db/ident :transfer
:db/doc "Data function that transfers an amount from one account to another."
:db/fn (eva/function
{:lang "clojure"
:params '[db from to amount]
:code '(let [from-balance (-> (d/entity db from) :account/balance (- amount))
to-balance (-> (d/entity db to) :account/balance (+ amount))]
[{:db/id from :account/balance from-balance}
{:db/id to :account/balance to-balance}])
})}])
@(eva/transact conn transfer)
Most of this is similar to the :inc-balance function we defined earlier, minus the addition of let. Let is a core concept of Clojure and allows us to define variables that are available only within the scope of the function. This helps clean up the code a bit, and obviously allows us to re-use values.
Now we can do this:
@(eva/transact conn [[:transfer
(eva/q get-entity-id db "Michael Jackson")
(eva/q get-entity-id db "Jeff Bridges")
5000]])
With this, we transfer 5000 from Michael Jackson's account into Jeff Bridge's account, as can be deduced from the result of the transaction.
Exceptions
Something we might want to do in a transaction function like this is to implement some logic to prevent a transfer from taking an account's balance into the negative.
(def transfer
[{:db/id (eva/tempid :db.part/user -1)
:db/ident :transfer
:db/doc "Data function that transfers an amount from one account to another."
:db/fn (eva/function
{:lang "clojure"
:params '[db from to amount]
:code '(let [from-entity (d/entity db from)
from-balance (-> from-entity :account/balance (- amount))
to-balance (-> (d/entity db to) :account/balance (+ amount))
]
(if (< from-balance 0)
(throw
(IllegalStateException.
(str "Transfer exception: Balance cannot be negative, current balance: "
(from-entity :account/balance))))
[{:db/id from :account/balance from-balance}
{:db/id to :account/balance to-balance}])
)})}
])
@(eva/transact conn transfer)
So what we've done here is introduce an if statement to our existing transfer function. (if (< from-balance 0)) will either throw an exception if the transfer would cause the balance of the account to become negative, or return the transactional data to make the transfer.
Now try this and see the exception:
@(eva/transact conn [[:transfer
(eva/q get-entity-id db "Michael Jackson")
(eva/q get-entity-id db "Jeff Bridges")
5001]])
Earlier, we decided to use the entity API to get the value of :account/balance for use inside our transaction function. We could have alternatively used the Datoms API, which I will demonstrate the use of now.
Datoms API
The Datoms API provides direct access to one of the datoms indexes. It takes an index name and a specification for a datom, returning an iterable of all datoms that match the spec.
Let's try it out:
(-> (eva/datoms db :eavt (eva/q get-entity-id db "Michael Jackson")))
You should see the representation for all datoms matching the entity id returned by our get-entity-id function. These datoms are retrieved directly from the :eavt index which we passed into the eva/datoms call, along with the db value. You should only see two datoms here, one for the :account/name and the other for the :account/balance.
We can narrow it down even further, by adding another specification to what we pass to the eva/datoms function.
(-> (eva/datoms db :eavt (eva/q get-entity-id db "Michael Jackson") :account/balance))
Now, we've specified that we want datoms that match the entity-id returned by get-entity-id and have an :account/balance attribute. Now we have the datom we are looking for but the value is still in the map, let's go get it.
(-> (eva/datoms db :eavt (eva/q get-entity-id db "Michael Jackson") :account/balance) first :v)
Just like magic. Basically first :v selects the first :v keyword in the map. :e, :a, :tx are bound in the same way and can also be referenced to get a particular value out of the datom map.
All of this begs the question, when would I want to use the datoms vs entity API. As a general rule of thumb, the datoms API is useful when you know precisely which index you want and which matching datoms you want to pull from said index. The entity API is useful when you know the entities you want, but don't necessarily know what attributes you care about, or you want to take advantage of lazy loading.
Also worth knowing is that we can call this transaction function directly, like so:
(let [transfer-function (-> transfer first :db/fn)
db-snapshot (eva/db conn)]
(transfer-function
db-snapshot
(eva/q get-entity-id db-snapshot "Michael Jackson")
(eva/q get-entity-id db-snapshot "Jeff Bridges")
100))
Let's go through this, step-by-step. (-> transfer first :db/fn) is selecting the :db/fn part of our transfer function. Try calling the function directly like this:
transfer
You will see the representation of the transaction function we defined before, including all the keywords :id, :ident, :doc, and :fn. So we assign the function part of the transaction function to transfer-function which allows us to call it as a function below.
(transfer-function db-snapshot (eva/q get-entity-id db-snapshot "Michael Jackson") (eva/q get-entity-id db-snapshot "Jeff Bridges") 100) is how we then call this function, passing the required arguments, as we did when we called this as part of the eva/transact operation.
Now let's get the latest datom as it exists in the database.
(def db (eva/db conn))
(eva/touch (eva/entity db (eva/q get-entity-id db "Jeff Bridges")))
Wait a second, the balance in the database still reports 100 but the result of calling transfer-function directly resulted in a balance of 200. We are about to make an important distinction. Earlier we used db-snapshot (eva/db conn) to get the latest value of the database. transfer-function was executed with that value as an argument and the result reflected the actual transfer operation. Without the call to eva/transact though, all we are doing when we call our transaction function is transforming a value.
Reified Transactions
We touched on transaction entities briefly in 101, now we are going to take a closer look. As you well know by this point, a transaction entity is created for every successful transaction. By default, the only attribute stored on the transaction entity is :db/txInstant, whose value is the time at which the transaction occurred. The transaction entity can be used for much more, let's look at an example:
(def audit-schema [
{:db/id (eva/tempid :db.part/db)
:db/ident :audit/from
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/doc "Origin account for transfer"
:db.install/_attribute :db.part/db}
{:db/id (eva/tempid :db.part/db)
:db/ident :audit/to
:db/cardinality :db.cardinality/one
:db/valueType :db.type/long
:db/doc "Destination account for transfer"
:db.install/_attribute :db.part/db}
{:db/id (eva/tempid :db.part/db)
:db/ident :audit/amount
:db/valueType :db.type/long
:db/cardinality :db.cardinality/one
:db/doc "Amount of transfer"
:db.install/_attribute :db.part/db}
{:db/id (eva/tempid :db.part/db)
:db/ident :audit/reason
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one
:db/doc "Reason for transfer"
:db.install/_attribute :db.part/db}
])
@(eva/transact conn audit-schema)
So we've transacted a few more attributes into the database that we can use to add metadata to transactions.
Next we'll modify our transfer database function to make use of this metadata.
(def transfer
[{:db/id (eva/tempid :db.part/user -1)
:db/ident :transfer
:db/doc "Data function that transfers an amount from one account to another."
:db/fn (eva/function
{:lang "clojure"
:params '[db from to amount reason]
:code '(let [from-entity (d/entity db from)
from-balance (-> from-entity :account/balance (- amount))
to-balance (-> (d/entity db to) :account/balance (+ amount))
]
(if (< from-balance 0)
(throw
(IllegalStateException.
(str "Transfer exception: Balance cannot be negative, current balance: "
(from-entity :account/balance))))
[{:db/id from :account/balance from-balance}
{:db/id to :account/balance to-balance}
{:db/id (d/tempid :db.part/tx) :audit/from from}
{:db/id (d/tempid :db.part/tx) :audit/to to}
{:db/id (d/tempid :db.part/tx) :audit/amount amount}
{:db/id (d/tempid :db.part/tx) :audit/reason reason}])
)})}
])
@(eva/transact conn transfer)
We've added the :audit attributes in our transfer transaction function. This will allow us to capture the from and to account entity id's for the transfer, as well as the amount and the reason for the transfer. Up until this point we've only dealt with :db.part/db when calling eva/tempid but here we call (d/tempid :db.part/tx). In this case we ask the transaction partition (:db.part/tx) for the temporary id that will be used for the transaction entity generated by this transaction and we can use that to associate other attributes with it.
Let's run some transactions that will actually make use of these new :audit attributes:
@(eva/transact conn [[:transfer
(eva/q get-entity-id db "Jeff Bridges")
(eva/q get-entity-id db "Michael Jackson")
1000
"Royalties."]])
@(eva/transact conn [[:transfer
(eva/q get-entity-id db "Jimmy Fallon")
(eva/q get-entity-id db "Michael Jackson")
500
"Obvious Money Laundering."]])
@(eva/transact conn [[:transfer
(eva/q get-entity-id db "Jimmy Fallon")
(eva/q get-entity-id db "Michael Jackson")
500
"Charity."]])
@(eva/transact conn [[:transfer
(eva/q get-entity-id db "Jeff Bridges")
(eva/q get-entity-id db "Michael Jackson")
2500
"Land transfer tax."]])
@(eva/transact conn [[:transfer
(eva/q get-entity-id db "Jeff Bridges")
(eva/q get-entity-id db "Michael Jackson")
1000
"Vast amounts of donuts."]])
Now that we have some useful transaction data, let's go get it:
(defn find-tx-ids [db account-id]
(eva/q '[:find [?tx-id ...]
:in $ ?account-id
:where
[?tx-id :audit/to ?account-id]]
db account-id))
find-tx-ids will query for all transaction entity ids where the passed in account-id matches the value for the :audit/to attribute.
(def db (eva/db conn))
(->> (eva/q get-entity-id db "Michael Jackson")
(find-tx-ids db)
(map (partial eva/entity db))
(map eva/touch)
(sort-by :db/txInstant))
Firstly, ->> is called a thread-last macro (we already talked about ->) which is used when you want to pass the result of a form as the last argument to the next form. Similarly to ->, using a threading macro makes this code much easier to read.
(eva/q get-entity-id db "Michael Jackson") gets us an entity id which we then pass as the last argument to (find-tx-ids db). That will give us all the transaction ids for this account which we then pass to (map (partial eva/entity db)). Couple things here, eva/entity takes a db value and also a single entity id. In this case we have a collection of entity ids potentially being returned by find-tx-ids so we need to use partial to wrap (eva/entity db) into a single function and then map so that we can apply each entity id to the partial function. We end up with a sequence of entities, which we then pass to eva/touch using the map function once again. This dereferences the entities in their entirety, and finally we call sort-by to order the entities by the :db/txInstant keyword. Refer to the section below for more information on sorting result sets in Eva.
Let's move on now to some API's and finally we will get to the historic functionality of the database.
Log API
Included within Eva is a database log of all transaction data in historic order. We can acquire a reference to the log like so:
(def log (eva/log conn))
Now using the tx-range function and supplying the log, we can see all the datoms asserted or retracted into the database between a specific start and end transaction. We can use eva/basis-t to get the most recent transaction number for a given db value.
(def db (eva/db conn))
(def datom-range (eva/tx-range log 0 (eva/basis-t db)))
(pprint datom-range)
Voilà, all the datoms from the beginning of the log to the latest transaction as reported by eva/basis-t returned as a list of maps. Inside each map are two keys, :t, containing the transaction number, and :data, containing the datoms asserted or retracted as part of that transaction.
So, for example, if we wanted to return a sequence of all the datoms we could do this:
(map :data datom-range)
Furthermore, you can filter a list of datoms to a specific entity like so:
(->> (nth (map :data datom-range) 2)
(filter (fn [datom] (= (:e datom) (eva/q get-entity-id db "Michael Jackson")))))
First we use nth to pull out the value of :data at a given index. More interesting is filter, which takes a predicate function and a collection as arguments, and filters out values which do not satisfy the predicate. In this case, (fn [datom] (= (:e datom) (eva/q get-entity-id db "Michael Jackson"))) defines an anonymous function that takes one argument, datom, and filters out values in our list where values for the key :e do not match the entity id returned by (eva/q get-entity-id db "Michael Jackson"). A short-form exists for writing an anonymous function, so we can replace what we wrote with (filter #(= (:e %) (eva/q get-entity-id db "Michael Jackson"))) and we will get the same result.
History Filters
Similar to the Clojure filter we just learned about, Eva itself has two functions that filter a database value based on a predicate. You can read more about them in detail here. We will go through each of them and show examples of how they are used below.
asOf
Unsurprisingly, (eva/as-of) allows us to see a database "as of" a particular point in time.
(def as-of-db (eva/as-of db 3))
So eva/as-of takes a db value and also a transaction number or entity id. eva/as-of then returns a filtered database excluding any datoms created beyond the supplied transaction identifier. So we can take this db value and use it just as we would any db value we've used up until this point. For example, we can find out the balance of an account holder as of the value of the database at this transaction.
(def ent (eva/entity as-of-db (eva/q get-entity-id db "Michael Jackson")))
(ent :account/balance)
Go ahead and try this for various different transaction ids to see the balance as it existed at different points in time.
History
The history view returns a value of the database which includes the present and all of the past. This view is excellent for querying for the full history of an entity, or group of entities. Important to note is that a database value acquired via history is incompatible with point-in-time functions (like entity) since it includes multiple values for the same fact at different times.
Get every account balance for a particular account:
(def history-db (eva/history db))
(eva/q '[:find ?balance
:in $
:where
[?e :account/name "Michael Jackson"]
[?e :account/balance ?balance]] history-db)
The above query will return all of the balances tied to the account of Michael Jackson. Try running this query again, but pass in db for history-db. When we pass in db to this query, we only get the latest balance associated with the account.
One thing you might notice is that historic query is not ordered based on when that balanace was asserted. In the case of the history of an account balance, order typically matters, so what can we do about that?
Sorting
Sorting is not implemented in Eva or Datalog. The only way to sort is by using built-in Clojure functions to sort an already existing result set.
(eva/q '[:find ?tx-id ?balance
:in $
:where
[?e :account/name "Michael Jackson"]
[?e :account/balance ?balance ?tx-id]] history-db)
Modifying the query to include the transaction id will give us something we can use to sort by, you might have noticed though that our result set seems to have doubled. We'll modify the query a bit, which will help to explain why that is the case.
(eva/q '[:find ?tx-id ?balance ?added
:in $
:where
[?e :account/name "Michael Jackson"]
[?e :account/balance ?balance ?tx-id ?added]] history-db)
Remember the last part of the datom 5-tuple is a boolean indicating whether the value was asserted or retracted. Let's modify the query once more.
(->> (eva/q '[:find ?tx-id ?balance ?added
:in $
:where
[?e :account/name "Michael Jackson"]
[?e :account/balance ?balance ?tx-id ?added]] history-db)
(sort-by first))
With the exception of the first transaction when no value had ever been asserted, everytime we assert a new value the old value is implictly retracted, which is what is being reflected here. So if we only want to know what the account balance was throughout history, we can adjust the query to only find values that were asserted.
(->> (eva/q '[:find ?tx-id ?balance
:in $
:where
[?e :account/name "Michael Jackson"]
[?e :account/balance ?balance ?tx-id true]] history-db)
(sort-by first))
We get a list of vectors containing transaction ids and balance values. Maybe we just want the balance though, easily achieved by doing this.
(->> (eva/q '[:find ?tx-id ?balance
:in $
:where
[?e :account/name "Michael Jackson"]
[?e :account/balance ?balance ?tx-id true]] history-db)
(sort-by first)
(map second))
Through the 101, and now the 102, we have covered a large swath of the features available in Eva. Up to this point we have been using the REPL extensively for demonstration purposes and really the only thing left is to tie this all together by building an actual application.
We hope you've enjoyed this tutorial, feel free to create an issue if you have any feedback.
Can you improve this documentation?Edit on GitHub
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close