mysql-queue.core — mysql-queue 1.1.0

A MySQL-backed durable queue implementation with scheduled jobs support.

Executable clj protocol

execute clj

(execute job db-conn log-fn err-fn timeout)

finished? clj

(finished? job)

Fertile clj protocol

beget clj

(beget parent)

(beget parent status)

(beget parent status parameters)

initialize! clj

(initialize! db-conn)

Create required MySQL tables if they don't exist. Returns true.

max-retries clj

Persistent clj protocol

cleanup clj

(cleanup entity conn)

persist clj

(persist entity conn)

schedule-job clj

(schedule-job db-conn name status params due-at)

Creates a scheduled job with provided name, status, args and due time. Returns the unique numeric id of a created job.

status clj

(status {:keys [db-conn fn-bindings]
         {recovery-threshold :recovery-threshold-mins} :options
         :as worker})

Returns a map describing the current status of the worker.

Stoppable clj protocol

stop clj

(stop worker timeout-secs)

ToDb clj protocol

*->db clj

(*->db entity)

ultimate-job-states clj

with-timeout clj macro

(with-timeout timeout & body)

A macro that executes the body in a future with a specified timeout. Returns nil if times out. Throws an exception if body returns nil.

worker clj

(worker db-conn
        fn-bindings
        &
        {:keys [buffer-size prefetch job-timeout-mins num-stats-jobs
                num-consumer-threads min-scheduler-sleep-interval
                max-scheduler-sleep-interval min-recovery-sleep-interval
                max-recovery-sleep-interval recovery-threshold-mins log-fn
                err-fn]
         :or {num-stats-jobs 50
              log-fn (constantly nil)
              prefetch 10
              min-scheduler-sleep-interval 0
              buffer-size 10
              max-scheduler-sleep-interval 10
              max-recovery-sleep-interval 10
              recovery-threshold-mins 20
              err-fn (constantly nil)
              job-timeout-mins 20
              num-consumer-threads 2
              min-recovery-sleep-interval 0}
         :as options})

Creates a new worker. Takes a database connection db-conn, a map of fn-bindings binding job names to job functions, and a number of optional keyword arguments:

• buffer-size - maximum number of jobs allowed into internal queue. Determines when the publisher will block. Default 10.
• job-timeout-mins - the number of minutes after which the job times out. Default 20.
• prefetch - the number of jobs a publisher fetches from the database at once. Default 10.
• num-stats-jobs - the number of jobs to keep in memory for statistical purpose. Per consumer thread. Default 50.
• num-consumer-threads - the number of concurrent threads that run jobs at the same time.
• min-scheduler-sleep-interval - the minimum time in seconds the scheduler will sleep before querying the database for due jobs. Default 0 seconds.
• max-scheduler-sleep-interval - the maximum time in seconds the scheduler will sleep before querying the database for due jobs. Default 10 seconds.
• min-recovery-sleep-interval - the minimum time in seconds the recovery thread will sleep before querying the database for stuck jobs. Default 0 seconds.
• max-recovery-sleep-interval - the maximum time in seconds the recovery thread will sleep before qerying the database for stuck jobs. Default 10 seconds.
• recovery-threshold-mins - the number of minutes after which a job is considered stuck and will be picked up by the recovery thread. Default 20.
• log-fn - user-provided logging function of 3 arguments: level (keyword), job (record), message (msg).
• err-fn - user-provided error function of one argument: error (Exception).