A background job processor
Backtick is a library for Clojure that pulls tasks from a queue and runs them. There is one central persistent queue for the jobs and the job runners can be distributed across multiple machines.
backtick artifacts are released to Clojars.
First configure your Postgres datastore by running
Backtick will read
:bt-database-url or failing that will use the
:database-url configuration parameter. For example:
You configure jobs using the
(require '[backtick.core :as bt])
(require '[clojure.tools.logging :as log])
(bt/define-worker log-sum [x y]
(log/infof "sum %s + %s = %s" x y (+ x y)))
and put the job in the queue by calling the
(schedule log-sum 3 4)
(schedule log-sum 88 99)
You may also run periodic jobs using
(bt/define-recurring heartbeat-every-5-minutes (* 1000 60 5) 
Or if you need you job to run at specific wall clock/calendar times, you may use
(bt/define-cron twice-in-the-afternoon "0 0 14,16 * * *" 
(log/info "do a thing"))
Backtick depends on the clj-cron-parse library to parse Cron specifications. The format is broadly the same as a standard crontab, with the addition of seconds in the first position. Briefly:
|day of month
||1-31 L W
||1-12 (or names)
|day of week
||0-7 (0 or 7 is Sun, or use names) W 1L 2L 3L 4L 5L 6L 7L
For more details on supported formats, consult clj-cron-parse's documentation.
define-cron creates a regular Clojure function of the same name.
You can run as many servers as you like to process the jobs. The server may be run using the start function:
or by running
backtick.core from a pre-built jar.
Backtick works by polling a Postgres table that acts as an incoming queue and then running jobs from that queue. There is one thread that polls Postgres and then hands the job to be processed by a separate thread pool.
Any data needed for the job is encoded as an EDN object in the Postgres table.
When a job is completed, it is marked in the queue as done. If the job fails, a recurring job will revive the dead one and place it back in the queue.
Polling isn't a great way to run a high volume queue, but using a Postgres table has the advantage of being simple, stable, and able to easily handle our load.
- Worker: A function that's registered with Backtick.
- Job: A unit of work that is scheduled to be run. Running a job causes a worker to be invoked.
- At job: A job scheduled to run at a particular time in the future.
- Recurring job: A job scheduled to run repeatedly after a particular time interval has elapsed.
- Cron job: A job that recurs based on the clock or calendar time.
- Runner: A process that executes a job.
Copyright (c) Chris Dean
Distributed under the Eclipse Public License either version 1.0 or (at your option) any later version.