firestore-clj

A Firestore API for Clojure. Provides tools for doing single pulls and writes, streaming real-time data, batched writes and transactions. This lib is a wrapper over com.google.firebase/firebase-admin. All functions are properly type hinted, so no reflection is used. We also try to provide somewhat idiomatic names for the operations and queries, and idiomatic transactions as well.

Getting started

You can use client-with-creds to get a client using credentials from a service account.

(require '[firestore-clj.core :as f])

(def db (f/client-with-creds "/path/to/creds.json"))

If you are using it inside Google Cloud Platform services with appropriate service account permissions, you can just provide the project-id using default-client:

(def db (f/default-client "project-id"))

Collections, documents, and subcollections

doc, docs, coll, colls and coll-group are the basic functiones here.

doc gets a reference for the doc with given path relative to its argument, or a reference to a new one if the argument is a collection and no path is given. coll gets a collection (relative to root) or subcollection (relative to a document).

(f/doc db "accounts/account1")
(f/doc (f/coll db "accounts") "account1") ; same as above
(f/doc db "accounts/account1/subcoll1/subdoc1") ; nesting is allowed
(f/doc (f/coll db "accounts")) ; reference to a new document with auto-generated id 

(f/coll db "accounts/account1/subcoll1")
(f/coll (f/doc db "accounts/account1/") "subcoll1") ; same as above

docs gets all documents of a collection, or maps over doc if a sequence of paths is given. colls gets all collections (relative to root) or subcolletions (relative to a document), or maps over coll if a sequence of paths is given

(f/docs (f/coll db "accounts")) ; all documents from accounts
(f/docs (f/coll db "accounts") ["account1" "account2"]) ; these two documents from accounts
(f/docs db ["accounts/account1" "accounts/account2"])

(f/colls db) ; all collections at root level
(f/colls (f/doc "accounts/account1")) ; all subcollections
(f/colls db ["accounts" "positions"])

coll-group returns a query including docs in all collections or subcollections with a given id.

(f/coll-group db "subcoll")

Writing data

We provide the methods add!, set!, create!, assoc!, dissoc!, merge! and delete!. Additionally, the functions server-timestamp, inc, mark-for-deletion, array-union and array-remove can be used as special values on a set!, merge! and assoc! operation. Some examples:

; creates new document with random id
(-> (f/coll db "accounts")
    (f/add! {"name"     "account-x"
             "exchange" "bitmex"}))

(def doc (-> (f/doc "accounts/xxxx")
             (f/set!{"name"        "account-x"
                     "exchange"    "bitmex"
                     "start_date"  (f/server-timestamp)}) ; creates doc (or overwrites it it already exists)
             (f/assoc! "trade_count" 0) ; updates one or more fields
             (f/merge! {"trade_count" (f/inc 1)
                        "active"      true}) ; updates one or more fields using a map
             (f/dissoc! "trade_count" "active"))) ; deletes fields

; deletes doc
(f/delete! doc)

Queries

We provide the query functions below (along with corresponding Java API methods):

firestore-clj	Java API
filter=	.whereEqualTo()
filter<	.whereLessThan()
filter<=	.whereLessThanOrEqualTo()
filter>	.whereGreaterThan()
filter>=	.whereGreaterThanOrEqualTo()
filter-in	.whereIn()
filter-contains	.whereArrayContains()
filter-contains-any	.whereArrayContainsAny()
start-at	.startAt()
start-after	.startAfter()
end-at	.endAt()
end-before	.endBefore()
select	.select()
sort-by	.orderBy()
limit	.limit()
offset	.offset()
range	.offset().limit()

Gotchas: limit and offset don't have the same semantics of take and drop. They are commutative, so both (-> q (t/offset 2) (t/limit 3)) and (-> q (t/limit 3) (t/offset 2)) will return query results at positions 2, 3, 4. Also, you can't chain multiple offsets and limits, only the last call to each is valid.

You can use pull to fetch the results as a map. Here's an example:

(-> (f/coll db "positions")
    (f/filter= "exchange" "bitmex")
    (f/limit 2)
    f/pull)

You can perform multiple equality filters using a map.

(-> (f/coll db "positions")
    (f/filter= {"exchange" "bitmex" 
                "account"  1}) 
    f/pull)

When result ordering matters, you can use pullv to get the results as vectors, or pullv-with-ids if you also need the ids.

(-> (f/coll db "positions")
    (f/filter= "account" 1)
    (f/sort-by "size") ; descending: (f/sort-by "size" :desc) 
    (f/start-at 10) ; ignore residual positions
    f/pullv) ; 

If you have the appropriate indexes, you can sort-by multiple fields:

(-> (f/coll db "positions")
    (f/filter= "account" 1)
    (f/sort-by "size" :desc "instrument") 
    f/pull)

Real-time data

You can materialize a document/collection reference or query as an atom with ->atom..., or stream updates as a Manifold stream with ->stream:

(def at (-> (f/coll db "positions")
            (f/filter= {"exchange" "bitmex" 
                        "account"  1}) 
            f/->atom))

(println @at)

; do stuff ...

(f/detach at) ; when you don't need updates anymore.

(require '[manifold.stream :as st])

(def stream (-> (f/coll db "positions")
                (f/filter= {"exchange" "bitmex" 
                            "account"  1}) 
                f/->stream))

(st/consume println stream)

(st/close! stream) ; when you don't need updates anymore.

Both ->atom and ->stream can also take a map with keys error-handler and a plain-fn that takes a snapshot and returns clojure data. Built-in plain-fns are ds->plain and ds->plain-with-id for document snapshots and qs->plain-map, qs->plainv and qs->plainv-with-ids for query snapshots. Default is snapshot->data, which uses ds->plain for documents and qs-plain-map for queries. Of course, you can pass identity if you just want the underlying snapshot.

If you need a lower level utility, you can use add-listener. It takes a 2-arity function and merely reifies it as an EventListener. The function changes might be useful: it takes a snapshot and generates a vector of changes, with :type, :reference, :new-index and :old-index keys. An example that just prints the ids of added, removed or modified docs.

(-> (f/coll db "accounts")
    (f/add-listener (fn [s e]
                      (doseq [{:keys [type reference]} (f/changes s)]
                        (case type
                          :added    (println "Added doc:" (f/id reference))
                          :modified (println "Modified doc:" (f/id reference))
                          :removed  (println "Deleted doc:" (f/id reference)))))))

Read upstream docs here for more.

Batched writes and transactions

The functions set, assoc, merge, dissoc, and delete are like their bang-ending counterparts, but merely describe operations to be done in a batched write/transaction context. They also return the batch/transaction itself, so you can easily chain operations. They are executed atomically by calling commit! or transact!.

(let [[acc1 acc2 acc3] (-> (f/coll db "accounts")
                           (f/docs ["acc1" "acc2" "acc3"]))]
  (-> (f/batch db)
      (f/assoc acc1 "tx_count" 0)
      (f/merge acc2 {"tx_count" 0})
      (f/delete acc3)
      (f/commit!)))

If you need reads, you'll need a transaction. Here's how you would transfer balances between two accounts:

(f/transact! db (fn [tx]
                  (let [[mine yours :as docs] (-> (f/coll db "accounts")
                                                  (f/docs ["my_account" "your_account"]))
                        [my-acc your-acc] (f/pull-docs docs tx)]
                    (f/set tx mine (-> (update my-acc "balance" + 100)
                                       (update "tx_count" inc)))
                    (f/set tx yours (-> (update your-acc "balance" - 100)
                                        (update "tx_count" inc))))))

You can use both pull and pull-docs in a transaction, passing the Transaction object as the second parameter.

Conveniences

We've also written a few convenience functions for common types of transactions and batches writes.

Updating a single field:

(f/update-field! (-> (f/coll db "accounts")
                     (f/doc "my_account"))
                 "balance" * 2)

Updating an entire doc

(f/update! (-> (f/coll db "accounts")
               (f/doc "my_account"))
           #(-> (update % "balance" * 2)
                (update "tx_count" inc)))

Updating many docs in a single transaction

Over a vector of document references:

(f/map! (-> (f/coll db "accounts")
            (f/docs ["my_account" "your_account"]))
        #(update % "balance" * 2))

Over results of a query:

(f/map! (-> (f/coll db "accounts")
            (f/filter< "balance" 1000))
        #(assoc % "balance" 1000))

Deleting multiple docs

In most cases delete-all! is enough. It accepts queries, including collections. It queries and writes in batches for efficiency.

(f/delete-all! (f/coll db "accounts"))
(f/delete-all! (-> (f/coll db "accounts")
                   (f/filter= "exchange" "deribit")))

However, it doesn't work if the queries contain limits or offsets, since they can't be chained and they are used internally for batching. In this case, use delete-all!*. It fetches all query results once and deletes in batches, therefore potentially consuming more memory for a while.

(f/delete-all!* (-> (f/coll db "accounts")
                    (f/limit 3)))

A more generic function is batch-delete!, which deletes an arbitrary seq of document references in batches. You can also use purge!, which deletes documents, collections and queries recursively.

Design decisions

• Many operations that were async by default on the Java API are sync here. That's mainly because you can't block ApiFutures with deref. If you want to go async, simply wrap with future.
• We assume all maps have string keys. We do not convert keywords. You can use camel-snake-kebab for doing conversions.

Contributing and improvements

We welcome PRs. Here are some things that need some work:

• Preconditions
• Set options
• More convenience around the objects returned from operations
• Investigate areTimestampsInSnapshotsEnabled to avoid conversions.

License

Copyright © 2020 Polvo Technologies.

Distributed under the MIT License.