plumbing.graph — prismatic/plumbing 0.6.0

A Graph is a simple, declarative way to define a composition of functions that is easy to define, modify, execute, test, and monitor.

This blog post provides a high-level overview of Graph and its benefits: http://plumatic.github.io/prismatics-graph-at-strange-loop

Concretely, a Graph specification is just a Clojure (nested) map with keyword keys and keyword functions at the leaves.

A Graph is defined recursively as either:

1. a keyword function (i.e., fn satisfying PFnk), or
2. a Clojure map from keywords to (sub)graphs.

A Graph is a declarative specification of a single keyword function that produces a map output, where each value in the output is produced by executing the corresponding keyword function in the Graph. The inputs to the keyword function are given by the outputs of other nodes in the graph with matching keywords (mimicking lexical scope in the case of nested maps), or failing that, from keywords in the input map.

For more details and examples of Graphs, see test/plumbing/graph_examples_test.cljx.

->graph clj/s

(->graph graph-nodes)

Convert a graph specification into a canonical well-formed 'graph', which is an array-map with nodes in a correct topological order that will respond to 'io-schemata' with a specification of the graph inputs and outputs.

The graph specification can be a Clojure map, in which case the topological order will be computed (an error will be thrown for cyclic specifications), or a sequence of key-value pairs that are already in a valid topological order (an error will be thrown if the order is not valid). Values in the input sequence are also converted to canonical graphs via recursive calls to ->graph.

check-comp-partial! clj/s

(check-comp-partial! g instance-fn)

Check that instance-fn is a valid fn to comp-partial with graph g.

comp-partial clj/s

(comp-partial g instance-fn)

Experimental.

An extension of pfnk/comp-partial that supplies new parameters to a subgraph, useful in composing hierarchical graphs.

g is a graph, and instance-fn is a fnk that takes arguments from the surrounding context and produces new parameters that are fed into g. Works by comp-partialing all leafs that expects any parameter produced by instance-fn with instance-fn, so beware of expensive instance-fns, or those that expect caching of some sort (i.e., attempt to generate shared state).

Throws an error if any parameter supplied by instance-fn is not used by at least one node in g.

comp-partial-fn clj/s

(comp-partial-fn f other)

Return a new pfnk representing the composition #(f (merge % (other %)))

compile clj/s

(compile g)

Compile graph specification g to a corresponding fnk using the a default compile strategy for host. Clojure: eager-compile ClojureScript: interpreted-eager-compile

eager-compile clj

(eager-compile g)

(eager-compile g
               {:keys [positional-limit]
                :or {positional-limit graph-positional/max-graph-size}
                :as _opts})

Compile graph specification g to a corresponding fnk that is optimized for speed. Wherever possible, fnks are called positionally, to reduce the overhead of creating and destructuring maps, and the return value is a record, which is much faster to create and access than a map. Compilation is relatively slow, however, due to internal calls to 'eval'.

Options: :positional-limit is used to decide when to switch to interpreted mode, which does not compile positionally. If positional compilation is required, use option {:positional-limit ##Inf} (at the risk of method-too-large errors).

graph clj/s

(graph & nodes)

An ordered constructor for graphs, which enforces that the Graph is provided in a valid topological ordering. This is a sanity check, and also enforces defining graphs in a readable way. Most explicit graphs should be created with this constructor.

(graph :x-plus-1 (fnk [x] (inc x)) :2-x-plus-2 (fnk [x-plus-1] (* 2 x-plus-1)))

in addition, an 'inline' graph can be provided in place of a key-value sequence, which will be merged into the graph at this position.

a named fnk can also be provided in place of a key-value pair, where the fnk's name (as a keyword) is the implicit key.

instance clj macro

(instance g m)

(instance g bind m)

Experimental.

Convenience macro for comp-partial, used to supply inline parameters to a subgraph (or fnk).

Example: (= {:x 21} (run (instance {:x (fnk [a] (inc a))} [z] {:a (* z 2)}) {:z 10}))

interpreted-eager-compile clj/s

(interpreted-eager-compile g)

Compile graph specification g to a corresponding fnk that returns an ordinary Clojure map of the node result fns on a given input. The compilation is much faster than 'eager-compile', but the compiled fn will typically be much slower.

io-schemata* clj/s

(io-schemata* g)

lazy-compile clj

(lazy-compile g)

Compile graph specification g to a corresponding fnk that returns a lazymap of the node result fns on a given input. This fnk returns the lazymap immediately, and node values are computed and cached as needed as values are extracted from the lazymap. Besides this lazy behavior, the lazymap can be used interchangeably with an ordinary Clojure map. Required inputs to the graph are checked lazily, so you can omit input keys not required by unneeded output keys.

par-compile clj

(par-compile g)

Experimental. Launches one future per node at startup; we probably woudln't use this in production, and will release more sophisticated parallel compilations later.

Compile graph specification g to a corresponding fnk that returns a lazymap of the node result fns on a given input. This fnk returns the lazymap immediately, and node values are computed and cached in parallel starting immediately (and attempts to extract values from the lazymap will block until each value is computed). Besides this lazy behavior, the lazymap can be used interchangeably with an ordinary Clojure map.

positional-eager-compile clj

(positional-eager-compile g arg-ks)

Like eager-compile, but produce a non-keyword function that can be called with args in the order provided by arg-ks, avoiding the overhead of creating and destructuring a top-level map. This can yield a substantially faster fn for Graphs with very computationally inexpensive node fnks.

Warning: if any level of g exceeds graph-positional/max-graph-size, compilation may fail. Do not use for arbitrarily large graphs.

profiled clj/s

(profiled profile-key g)

Modify graph spec g, producing a new graph spec with a new top-level key 'profile-key'. After each node value is computed, the number of milliseconds taken to compute its value will be stored under an atom at 'profile-key'.

restricted-call clj/s

(restricted-call f in-m)

Call fnk f on the subset of keys its input schema explicitly asks for

run clj/s

(run g input)

Eagerly run a graph on an input by compiling and then executing on this input.

simple-flat-compile clj/s

(simple-flat-compile g check-input? make-map assoc-f)

Helper method for simple (non-nested) graph compilations that convert a graph specification to a fnk that returns a Clojure map of the graph node values. (make-map m) converts an initial Clojure map m to the return type of the fnk, and (assoc-f m k f) associates the value given by (f) under key k to map m.

simple-hierarchical-compile clj/s

(simple-hierarchical-compile g check-input? make-map assoc-f)

Hierarchical extension of simple-nonhierarchical-compile.

working-array-map clj/s

(working-array-map & args)

array-map in cljs no longer preserves ordering, replicate the old functionality.