reflet.db — io.zalky/reflet 0.3.0-rc1

cljs

Provides an event sourced, reactive db, with mutation and query methods.

Storing data in graph form has many benefits for application and data design. The main trade-off is the performance cost of normalizing data during writes, and de-normalizing data during queries. Specifically, some graph queries can end up spanning the entire db, and so in a reactive application, understanding when to re-run queries is key to performance.

The naive approach of just re-running every query whenever the db changes is rarely feasible. Just one expensive query will grind the entire app to a halt, and predicting the cost of a query is a run-time calculation that depends on the data.

A key insight that can help optimize query denormalization is that g for the set of queries that conform to EQL, a result cannot change if the entities that were traversed while walking the joins have not changed.

An approach used elsewhere that leverages this guarantee is to use reagent reactions to cache entity joins in the computed result. This avoids recomputing the entire query if some of the entities that were traversed in the joins have not changed. Only the subgraph that depends on the changed entities will be recomputed.

Unfortunately this approach runs into a fundamental problem: by turning every join into a reactive computation, the cumulative overhead of tracking every reaction significantly exceeds the cost of just running the full non-reactive query. Equality checks are expensive, and the number of join reactions created depends on the shape of data at run-time, which can be huge. The naive approach, to just recompute the full query when the db changes, is always faster.

The solution presented in this namespace leverages the guarantee on query results and combines it with an event sourcing strategy to implement a differential, reactive loop between the db and queries.

To summarize: each query tracks the entities it traversed to produce its result. When a mutation operation is performed on the db, the entities that were touched by the mutation are then directly mapped to the queries that need to update. This is essentially a mutation event sourcing strategy.

Importantly, while queries react to mutation events, they will also always return the correct value if for whatever reason the db is reset outside the mutation functions, for example, if the db value is set by Re-frame-undo or debug tooling like Re-frame-10x. The reactive queries respect time-travel.

Another key design point is that query operations must be commutative and idempotent with respect to the tracking index. On the mutation side, any set of db operations that were previously commutative or idempotent on non-graph data should remain so.

The db mutation operations are pure functions that operate on the db value provided to event handlers. They can be arbitrarily composed in event handler code, similar to associative methods like clojure.core/assoc. They can also be freely interleaved with non-normalzied db operations. At the end of the day the Re-frame db is still just a big map.

It is important to note that the query index is not actually stored as part of application state, since it is fundamentally not application state. Rather, it is state associated with the reactive queries, analogous to how reagent reactions maintain internal state to function properly. If you wind back application state to a previous value, the query state should not change until the queries have re-run.

In order for the pure db mutation functions to operate on the index, an interceptor injects the index into the db coeffect, as well as removes and commits the updated index as part of the :after interceptor chain.

Three iteration ticks drive the reactive algorithm:

1. db tick
2. index tick
3. query tick one for each query

Step by step:

1. The db tick and index tick are incremented synchronously by the database mutation functions. If at any point the db tick and index tick do not match, this indicates time travel, and the query index is flushed by the mutation function.

2. If an entity is touched by a db mutation function, then any query that was tracking that entity is marked as touched in the query index.

3. During the subscription phase, a query must be re-run iff one or more of the following is true:

   a) the query is marked as touched in the query index, b) the query's tick is nil, indicating a fresh query index, or c) the db tick is not equal to the index tick, indicating time travel

4. This conditional constitutes the ::query-tick signal. The signal returns the current query tick if the query does not need to re-run, or the current db tick if it does.

5. When a query is re-run, it: a) updates the entities it is tracking in the query index b) clears the touched query set c) syncs the query tick to the db tick

6. After all queries have been run, the index tick is synced to to the db tick.

The above algorithm will only react to db changes via the db mutation functions. Specifically, the graph data stored at the ::data key of the Re-frame db should never be directly manipulated, except via the mutation functions.

However, aside from this one constraint, all regular db operation on non-graph data are completely orthogonal to the differential reactive loop. At the end of the day, the db is still just a map, and you can mutate it in any way as long as you persist the ::data key.

This namespace additionally provides an implementation for link queries similar to those in Fulcro or Om Next. A link attribute allows you to store graph data at a semantically meaningful, global keyword in the graph db. For example, you could store graph user data at a global ::current-user attribute in the db index.

Finally, this algorithm is implemented entirely via existing reagent and Re-frame machinery.

assoc-inn cljs

(assoc-inn db [[a v :as ref] :as path] value)

Adds the unnormalized value to entity at path. Similar semantics to assoc-in, where the entity ref is first part of path. Additionally touches any graph queries that are tracking the entity and increments the db and index ticks. Does not resolve entity references, so it cannot do deep updates in graph data.

assocn cljs

(assocn db attr tx)

Adds a normalized link at the given attribute.

default-pull-impl cljs

(default-pull-impl {ref :ref :as context} expr)

Evaluates the pull expression, expr, against the :db given in the context map. context will contain at least:

:db DB value against which the expression is evaluated

:id-attrs A set of unique id attributes that represent joins to other entities.

And optionally:

:ref Root entity reference with which to start the graph traversal. If it is omitted, the query is considered to be a link query. [Optional]

:acc-fn Fn that accumulates touched entity references via side effects. Required for reactivity. [Optional]

:pull-fx-fn Fn that handles pull side effects, like triggering data syncs. [Optional]

After evaluating the expression against the value of the db, this function returns the result.

By default, this implementation is a pure function of the given :db value. However, if :acc-fn and :pull-fx-fn are provided in the context map, this implementation runs them for side-effects. This means the entire impl must also be eager. This function is not meant to be called directly.

default-unique-attributes cljs

dissocn cljs

(dissocn db & tx)

Removes the list of entities, refs, or links from the db, touches any queries tracking those entities in the index, and increments the db and index ticks. Does not remove any nested entities in tx.

get-inn cljs

(get-inn db [ref :as path] & [or])

Pulls graph data from the given db at the given path. First element in path should be an entity ref. Uses get-in semantics. Does not resolve references. This is a non-reactive fn meant to be used in event handlers.

get-pull-fn cljs

(get-pull-fn)

Retrieves configured pull fn, or default implementation.

getn cljs

(getn db ref & [or])

Pulls the graph entity from the db at the given path. Uses get semantics. This is a non-reactive fn meant to be used in event handlers.

inject-query-index cljs

Injects query index into app state for graph data fns to work.

mergen cljs

(mergen db tx)

Normalizes the given tx data, merges all normalized entities into the graph db, touches any queries tracking those entities in the index, and increments the db and index ticks.

mount-ref! cljs

(mount-ref! ref)

mounted? cljs

(mounted? ref)

Returns true if the given entity references is associated with a component that is currently mounted. This is useful for long running tasks that should short circuit if the associated component has been unmounted.

new-db cljs

(new-db)

(new-db data)

(new-db data id-attrs)

Returns a new db, optionally with initial data and unique id attributes. Must have at least one unique id attribute.

prop? cljs

(prop? expr)

pull cljs

(pull db expr)

(pull db expr e-ref)

Pulls graph data from the given db. This is a non-reactive, functionally pure version of pull-reactive for use in event handlers. The data is pulled from the entity according to a pull pattern that conforms to:

{ :attr sub-pattern } If the entity contains :attr, it is treated as an entity reference and resolved. The algorithm then pulls the sub-pattern from the resolved entity.

[ sub-pattern-1 sub-pattern-2 sub-pattern-3 ] The algorithm attempts to pull each sub-pattern from the given entity, and merging all the results together.

keyword If the entity contains the keyword, includes it in the entity result.

'* (literal symbol asterisk) Includes all attributes from the entity in the result.

{ :attr '... } Similar to the map pattern, but is recursive where the sub-pattern is the same as the parent entity reference and resolved. The algorithm then pulls the sub-pattern from the resolved entity. The ellipses can be replaced with an natural number to specify a limit to the recursion.

pull-fx cljs multimethod

pull-reaction cljs

(pull-reaction config expr-fn query-v)

Returns a differential pull reaction. expr-fn is a function that given query vector arguments, returns a pull query spec and optionally an entity ref that is being queried. query-v is the query vector supplied to the generating subscription. The tracing of this reaction was patterned after the re-frame.sub/reg-sub implementation, and allows subscriptions based on these reactions to be inspected in re-frame-10x. Note that while the computation in the pull reaction is dependent on the db and query index values, it is not reactive to them. Instead, it is reactive to changes the query tick, which tracks the db-tick and synced.

query-index cljs

query-index-fx cljs

queue-size cljs

(queue-size)

Number of events to tap per query. This should eventually be dynamic.

random-ref cljs

(random-ref id-attr & [meta])

Given a unique id attribute, and optionally metadata, returns a random entity reference. Only rebound for testing.

random-ref-impl cljs multimethod

Extend this to produce different kinds of random entity references for new id attributes.

ref-meta cljs

(ref-meta ref)

Returns the metadata on an entity reference

result-reaction cljs

(result-reaction input-r result-fn query-v)

tap-fn cljs

trace-event cljs

trace-index cljs

trace? cljs

(trace? v)

traced-reaction clj/s macro

(traced-reaction q-ref query-v reaction-fn & [dispose-fn])

transient-unmounted? cljs

(transient-unmounted? ref)

Returns true if the given reference is unmounted and transient.

transient? cljs

(transient? ref)

Returns true if the given entity references is transient.

unmount-ref! cljs

(unmount-ref! ref)

update-inn cljs

(update-inn db [[a v :as ref] :as path] f & args)

Updates the value at the graph db path. Similar semantics to clojure.core/update-in, where the entity ref or link attribute is first part of path. The updated value is either an entity, a link attribute value, or an attribute of an entity. Does not resolve entity references, so it cannot do deep updates in graph data.

updaten cljs

(updaten db attr f & refs)

Updates a normalized link at the given attribute.