superlifter

Superlifter is an implementation of DataLoader for Clojure.

To quote from the DataLoader readme:

DataLoader allows you to decouple unrelated parts of your application without sacrificing the performance of batch data-loading. While the loader presents an API that loads individual values, all concurrent requests will be coalesced and presented to your batch loading function. This allows your application to safely distribute data fetching requirements throughout your application and maintain minimal outgoing data requests.

Superlifter uses Urania, a remote data access library for Clojure/script inspired by Haxl which in turn inspired DataLoader. Urania allows batching of similar fetches and deduplication via caching of identical fetches.

Superlifter adds smooth integration with libraries like lacinia, where GraphQL resolvers are run independently and must return data (or promises of data), leading to 1+n problems which can otherwise only be resolved by prefetching which complicates code.

The aim of superlifter is to provide a way of combining fetches delineated by time buckets, thresholds or explicit trigger rather than by node resolution.

As the underlying fetches are performed by Urania, knowledge of this library is required (it's very simple, though!).

Superlifter provides the following features:

• Fast, simple implementation of DataLoader pattern
• Bucketing by time or by queue size
• Asynchronous fetching
• Batching of fetches
• Shared cache for all fetches in a session
  • Guarantees consistent results
  • Avoids duplicating work
• Access to the cache allows longer-term persistence

• Vanilla usage
• Lacinia usage

Vanilla usage

Start a superlifter as follows:

(require '[superlifter.api :as s])
(require '[urania.core :as u])

(def context (s/start! {:buckets {:default {:triggers {}}}}))

This superlifter has no triggers, and must be fetched manually. Other kinds of trigger include queue-size and interval (like DataLoader), detailed below. Remember to call (s/stop! context) when you have finished using it.

You can enqueue items for fetching:

(s/with-superlifter context
  (def hello-promise (s/enqueue! (u/value "Hello world"))))

When the fetch is triggered the promises will be delivered.

Triggering fetches

Regardless of the trigger used, you can always manually trigger a fetch of whatever is currently in the queue using (s/with-superlifter context (s/fetch!)). This returns a promise which is delivered when all the fetches in the queue are complete, containing the results of all the fetches.

On demand

In the example above no triggers were specified. Fetches will only happen when you call (s/with-superlifter context (s/fetch!)).

Queue size trigger

You can specify that the queue is fetched when the queue reaches a certain size. You can configure this to e.g. 10 using the following options:

{:triggers {:queue-size {:threshold 10}}}

Interval trigger

You can specify that the queue is fetched every e.g. 100ms using the following options:

{:triggers {:interval {:interval 100}}}

This will give batching by time in a similar fashion to DataLoader.

Your own trigger

You can register your own kind of trigger by participating the in s/start-trigger! multimethod, so you can listen for other kinds of events that might let you know when it's a good time to perform the fetch. See the interval trigger implementation for inspiration.

Trigger combinations

You can supply any number of triggers which will all run concurrently and the queue will be fetched when any one condition is met.

{:triggers {:queue-size {:threshold 10}
            :interval {:interval 100}}}

Lacinia usage

Given the following schema in lacinia:

(def schema
 {:objects {:PetDetails {:fields {:name {:type 'String}
                                  :age {:type 'Int}}}
            :Pet {:fields {:id {:type 'String}
                           :details {:type :PetDetails
                                     :resolve resolve-pet-details}}}}
  :queries {:pets
            {:type '(list :Pet)
             :resolve resolve-pets}}})

Where the resolvers are as follows:

(defn- resolve-pets [context args parent]
  (let [ids (keys (:db context))]
    (map (fn [id] {:id id}) ids)))

;; invoked n times, once for every id from the parent resolver
(defn- resolve-pet-details [context args {:keys [id]}]
  (get-in (:db context) id))

We can rewrite this using superlifter (see the example code for full context):

;; superlifter.lacinia has a different `with-superlifter` macro
;; to help you return a lacinia promise
(require '[superlifter.lacinia :refer [with-superlifter]])
(require '[clojure.tools.logging :as log])

;; def-fetcher - a convenience macro like defrecord for things which cannot be combined
(s/def-fetcher FetchPets []
  (fn [_this env]
    (map (fn [id] {:id id}) (keys (:db env)))))

;; def-superfetcher - a convenience macro like defrecord for combinable things
(s/def-superfetcher FetchPet [id]
  (fn [many env]
    (log/info "Combining request for" (count many) "pets")
    (map (:db env) (map :id many))))

(defn- resolve-pets [context _args _parent]
  (with-superlifter context
    (-> (s/enqueue! (->FetchPets))
        (s/add-bucket! :pet-details
                        (fn [pet-ids]
                          {:triggers {:queue-size {:threshold (count pet-ids)}
                                      :interval {:interval 50}}})))))

(defn- resolve-pet-details [context _args {:keys [id]}]
  (with-superlifter context
    (s/enqueue! :pet-details (->FetchPet id))))

It's usual to start a Superlifter before each query and stop it afterwards. There is an inject-superlifter interceptor which will help you do this:

(require '[com.walmartlabs.lacinia.pedestal :as lacinia])
(require '[com.walmartlabs.lacinia.schema :as schema])
(require '[superlifter.lacinia :refer [inject-superlifter]])

(def pet-db (atom {"abc-123" {:name "Lyra"
                              :age 11}
                   "def-234" {:name "Pantalaimon"
                              :age 11}
                   "ghi-345" {:name "Iorek"
                              :age 41}}))

(def lacinia-opts {:graphiql true})

(def superlifter-args
  {:buckets {:default {:triggers {:queue-size {:threshold 1}}}}
   :urania-opts {:env {:db @pet-db}}})

(def service
  (lacinia/service-map
   (fn [] (schema/compile schema))
   (assoc lacinia-opts
          :interceptors (into [(inject-superlifter superlifter-args)]
                              (lacinia/default-interceptors (fn [] (schema/compile schema)) lacinia-opts)))))

Build

License

Copyright © 2019 oliyh

Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.