overseer.api
User-facing core API
Jobs are defined as maps with the following attributes: :job/id - Required String uniquely identifying a job :job/type - Keyword type name of this job :job/status - Keyword, one of #{:unstarted :started :finished :failed :aborted} :job/args - Optional arguments to the job (must be serializable as EDN)
Jobs that have been run (or attempted) by the system may also have the following attributes: :job/heartbeat - Integer UNIX timestamp of the last time a worker processing this job marked itself alive :job/failure - Map containing information about the failure of a job
Graphs are defined as Loom digraphs where every node is a valid Job.
User-facing core API
Jobs are defined as maps with the following attributes:
:job/id - Required String uniquely identifying a job
:job/type - Keyword type name of this job
:job/status - Keyword, one of #{:unstarted :started :finished :failed :aborted}
:job/args - Optional arguments to the job (must be serializable as EDN)
Jobs that have been run (or attempted) by the system may also have the following attributes:
:job/heartbeat - Integer UNIX timestamp of the last time a worker processing
this job marked itself alive
:job/failure - Map containing information about the failure of a job
Graphs are defined as Loom digraphs where every node is a valid Job.abortclj
(abort msg)Control-flow helper to immediately mark a job as aborted from within a handler
and abort all of its dependents (halts handler execution)
Fails loudly by default - will log an error and and send to Sentry.
See also abort-silent
Control-flow helper to immediately mark a job as aborted from within a handler and abort all of its dependents (halts handler execution) Fails loudly by default - will log an error and and send to Sentry. See also `abort-silent`
abort-silentclj
(abort-silent msg)Like abort, but does not send exceptions to Sentry
Like `abort`, but does not send exceptions to Sentry
faultclj
(fault msg)Signal that a transient fault has occurred and the worker should release and unstart the job so that it can be retried at a later time.
Signal that a transient fault has occurred and the worker should release and unstart the job so that it can be retried at a later time.
harnessclj
(harness handler wrapper)(harness handler k wrapper)A mechanism to 'wrap' job handlers, giving one the ability to provide additional context, inject dependencies, or otherwise alter the flow of execution. Conceptually, harnessing is similar to the idea behind Ring middleware.
Accepts a standard job handler (map or function) and a
wrapper function which will be called with the function specified
in your handler and is expected to return a new function with the
same signature. If your handler is a map, it will be transparently
constructed/deconstructed; harnesses work solely in terms of functions.
For example, a harness that simply implements the default behavior for a processor is:
(defn my-harness [f] (fn [job] (f job)))
A more substantive harness can be used to provide jobs with additional context or inject dependencies such as a database connection:
(defn my-harness [f] (fn [job] (let [modified-job (assoc job :conn (d/connect my-datomic-uri))] (f modified-job))))
Or add logging:
(defn logging-harness [f] (fn [job] (println "START execution of " (:job/id job)) (f job) (println "FINISH execution of " (:job/id job))))
After defining a harness, in the job-handlers map one specifies
{:my-job (overseer.api/harness my-job/run my-harness)}
Following the previous example, within your handler you now have additional context available:
(defn run [{:keys [conn] :as job}] ...)
If your handler is a map, you can optionally specify a key to harness a specific stage; the default is :process. For example, to harness a post-processor:
{:my-job (overseer.api/harness my-job/run :post-process my-harness)}
If you attempt to harness a missing stage for a given job, the wrapper will be invoked with a properly-formed identity function, meaning you can write your harnesses in a single consistent fashion, and, for example, universally harness a post-processor for a set of handlers that may or may not define their own post-processor.
A mechanism to 'wrap' job handlers, giving one the ability
to provide additional context, inject dependencies, or otherwise
alter the flow of execution. Conceptually, harnessing is similar to
the idea behind Ring middleware.
Accepts a standard job handler (map or function) and a
`wrapper` function which will be called with the *function* specified
in your handler and is expected to return a new *function* with the
same signature. If your handler is a map, it will be transparently
constructed/deconstructed; harnesses work solely in terms of functions.
For example, a harness that simply implements the default behavior for
a processor is:
(defn my-harness [f]
(fn [job]
(f job)))
A more substantive harness can be used to provide jobs with additional
context or inject dependencies such as a database connection:
(defn my-harness [f]
(fn [job]
(let [modified-job (assoc job :conn (d/connect my-datomic-uri))]
(f modified-job))))
Or add logging:
(defn logging-harness [f]
(fn [job]
(println "START execution of " (:job/id job))
(f job)
(println "FINISH execution of " (:job/id job))))
After defining a harness, in the job-handlers map one specifies
{:my-job (overseer.api/harness my-job/run my-harness)}
Following the previous example, within your handler you now have
additional context available:
(defn run [{:keys [conn] :as job}] ...)
If your handler is a map, you can optionally specify a key to harness a
specific stage; the default is :process. For example, to harness
a post-processor:
{:my-job (overseer.api/harness my-job/run :post-process my-harness)}
If you attempt to harness a missing stage for a given job, the wrapper will
be invoked with a properly-formed identity function, meaning you can write
your harnesses in a single consistent fashion, and, for example, universally
harness a post-processor for a set of handlers that may or may not define
their own post-processor.installclj
(install store)Install store configuration (create system tables, etc). Not guaranteed to be idempotent. Returns :ok on success
Install store configuration (create system tables, etc). *Not* guaranteed to be idempotent. Returns :ok on success
job-graphclj
(job-graph job-type-graph)(job-graph job-type-graph tx)Given a map in Loom adjacency list format specifying dependencies between keyword job types, and an optional map of additional job data, return a Loom digraph of Job maps that can be transacted by a store. This assumes you are only generating one job per type.
Ex: (def graph (job-graph {:start [] :process-1a [:start] :process-1b [:process-1a] :process-2 [:start] :finish [:process-1b :process-2]} {:org/id 123})) ; => Graph<Job>, ex: ; {{:job/id 1 :job/type :start :org/id 123} [] {:job/id 2 :job/type :process-1a :org/id 123} [{:job/id 1 ...}] {:job/id 3 :job/type :process-1b :org/id 123} [{:job/id 2 ...}] ...} (core/transact-graph store txns graph)
Given a map in Loom adjacency list format specifying dependencies between keyword job types,
and an optional map of additional job data, return a Loom digraph of Job maps that can be
transacted by a store. This assumes you are only generating one job per type.
Ex:
(def graph (job-graph
{:start []
:process-1a [:start]
:process-1b [:process-1a]
:process-2 [:start]
:finish [:process-1b :process-2]}
{:org/id 123}))
; => Graph<Job>, ex:
; {{:job/id 1 :job/type :start :org/id 123} []
{:job/id 2 :job/type :process-1a :org/id 123} [{:job/id 1 ...}]
{:job/id 3 :job/type :process-1b :org/id 123} [{:job/id 2 ...}]
...}
(core/transact-graph store txns graph)simple-graphclj
(simple-graph & jobs)Construct a Graph from Job(s) that have no dependencies between them
Construct a Graph from Job(s) that have no dependencies between them
startclj
(start config store job-handlers)Start the system inline given a config map, a Store implementation (see store)
and a job-handler map of {job-type job-handler}
Start the system inline given a config map, a Store implementation (see `store`)
and a job-handler map of {job-type job-handler}storeclj
(store config)Return a Store implementation based on the store type and type-specific
configuration in config map
Return a Store implementation based on the store type and type-specific configuration in `config` map
transact-graphclj
(transact-graph store graph)Given a Graph, atomically transact all of its jobs/dependencies into
the store. See store, job-graph.
Given a Graph, atomically transact all of its jobs/dependencies into the store. See `store`, `job-graph`.
validate-graph-handlersclj
(validate-graph-handlers handlers job-type-graph)Assert that a given job-type keyword graph (see job-graph)
only references handlers defined in the given handlers map.
Assert that a given job-type keyword graph (see `job-graph`) only references handlers defined in the given `handlers` map.
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close