How the :db and :redis queues store a job: transit, so a handler gets back the keywords, UUIDs and instants it was given, as the in-memory queue does. They stored JSON until BOU-689, and jobs written then still decode.
How the :db and :redis queues store a job: transit, so a handler gets back the keywords, UUIDs and instants it was given, as the in-memory queue does. They stored JSON until BOU-689, and jobs written then still decode.
Database-backed job queue (IJobQueue) on next.jdbc.
Lets an app run durable background jobs on its existing SQL database (H2, PostgreSQL or MySQL) without standing up Redis, and survive a Redis outage. Implements the same reliable-dequeue contract as the Redis adapter:
dequeue-job! atomically claims the next ready row (sets status=processing,
locked_by/locked_at) so a worker that crashes mid-job does not lose it.ack-job! removes the row once the worker is done with it.reclaim-abandoned-jobs! returns rows whose lease has expired
(locked_at older than lease-ms) back to ready, i.e. at-least-once.Portable SQL only (no SELECT ... FOR UPDATE SKIP LOCKED): the claim is a
SELECT of the best candidate followed by a conditional UPDATE ... WHERE status='ready'. Concurrent workers that pick the same row have their UPDATEs
serialized by the row lock — exactly one wins (update count 1); the losers
retry the next candidate. This runs identically on H2, PostgreSQL and MySQL, so the
reliability tests run on the default in-memory H2 test database.
The authoritative job value is the transit payload column (same wire format
as the Redis adapter, wagoe.jobs.shell.adapters.codec). The
indexed columns (queue, priority_rank, status, execute_at, locked_at) exist
only for ordering, claiming, and reclaim.
The worker renews the lease while a handler runs (ports/ILeasedJobQueue),
so lease-ms bounds how long a crashed worker's job waits, not how long a
job may run (BOU-823).
Caveats:
payload is VARCHAR(1000000) (LONGTEXT on MySQL); keep serialized
jobs under ~1 MB.Database-backed job queue (IJobQueue) on next.jdbc. Lets an app run durable background jobs on its existing SQL database (H2, PostgreSQL or MySQL) without standing up Redis, and survive a Redis outage. Implements the same reliable-dequeue contract as the Redis adapter: - `dequeue-job!` atomically claims the next ready row (sets status=processing, locked_by/locked_at) so a worker that crashes mid-job does not lose it. - `ack-job!` removes the row once the worker is done with it. - `reclaim-abandoned-jobs!` returns rows whose lease has expired (locked_at older than `lease-ms`) back to `ready`, i.e. at-least-once. Portable SQL only (no `SELECT ... FOR UPDATE SKIP LOCKED`): the claim is a SELECT of the best candidate followed by a conditional `UPDATE ... WHERE status='ready'`. Concurrent workers that pick the same row have their UPDATEs serialized by the row lock — exactly one wins (update count 1); the losers retry the next candidate. This runs identically on H2, PostgreSQL and MySQL, so the reliability tests run on the default in-memory H2 test database. The authoritative job value is the transit `payload` column (same wire format as the Redis adapter, `wagoe.jobs.shell.adapters.codec`). The indexed columns (queue, priority_rank, status, execute_at, locked_at) exist only for ordering, claiming, and reclaim. The worker renews the lease while a handler runs (`ports/ILeasedJobQueue`), so `lease-ms` bounds how long a crashed worker's job waits, not how long a job may run (BOU-823). Caveats: - `payload` is `VARCHAR(1000000)` (`LONGTEXT` on MySQL); keep serialized jobs under ~1 MB.
In-memory job queue implementation for development and testing.
Uses Clojure atoms and data structures for job queuing. This adapter is suitable for:
NOT suitable for:
State is stored in atoms with the following structure:
In-memory job queue implementation for development and testing. Uses Clojure atoms and data structures for job queuing. This adapter is suitable for: - Local development without Redis - Fast unit testing - CI/CD pipelines - Learning and tutorials NOT suitable for: - Production use (no persistence, single-process only) - Distributed systems (not shared across processes) - High-volume job processing (limited by memory) State is stored in atoms with the following structure: - jobs: Map of job-id -> job - queues: Map of queue-name -> priority-map of [priority job-ids] - scheduled: Sorted set of [execute-at job-id] pairs - failed: Vector of failed job-ids - stats: Map of queue-name -> statistics
Redis-backed job queue implementation.
Uses Redis for distributed job queuing with the following Redis data structures:
This adapter provides production-grade job queuing with:
Redis-backed job queue implementation. Uses Redis for distributed job queuing with the following Redis data structures: - Sorted Sets: For scheduled jobs (scored by execute-at timestamp) - Lists: For priority queues (critical, high, normal, low) - Hashes: For job data storage - Sets: For tracking workers This adapter provides production-grade job queuing with: - Distributed queue across multiple workers - Priority-based job processing - Scheduled job execution - Job persistence - Atomic operations
Integrant wiring for the jobs module.
Config keys:
:wagoe/jobs The settings block that switches the module on: {:provider :memory | :db | :redis ; default :memory :lease-ms 60000 ; :db only, in-flight lease :redis {:host "localhost" :port 6379} :workers {:count 1 :queue-name :default :virtual-threads? false} :recurring {:nightly-report {:cron "0 2 * * *" :zone "Europe/Amsterdam"}} :scheduler {:tick-ms 5000 :lease-ms 30000 :zone "UTC"} :on-outcome my.app.jobs/on-outcome ; optional, (fn [outcome]) :report-failures-every-ms 3600000 ; error-tracker throttle :expect-success {:nightly-report 26}} ; job type -> hours, else reported
:wagoe/jobs-runtime Queue, store and stats from one adapter, so the three share a backend.
:wagoe/job-queue, :wagoe/job-store, :wagoe/job-stats
What other modules and the devtools dashboard take a ref to — workflow
and push already document :job-queue. Projections of the runtime rather
than components of their own, because the in-memory adapter's queue and
store must share state. :wagoe/job-stats is nil under :provider :db,
which ships no IJobStats.
:wagoe/idempotency-store
IIdempotencyStore on the runtime's backend, for any module.
:wagoe/job-scheduler
The recurring-job ticker: :recurring {job-type {:cron "0 2 * * *" …}}
from the settings block, plus whatever recurring/schedule-recurring!
adds. :scheduler {:tick-ms :lease-ms :zone :watch-ms} tunes it. With
:expect-success its lease holder reports a job type that has not
succeeded within its hours (outcomes/check-stale!).
:wagoe/job-registry
The handler map every module contributed. Modules contribute handlers the
way they contribute routes: a :job-handlers vector in their ig-config,
collected by wagoe.platform.shell.system.config (BOU-330 pattern).
:wagoe/job-workers
The worker pool. :count 0 builds none — that is a web node in the
web/worker split; the default of 1 is a single process that both enqueues
and runs, which is what an application without a deployment topology has.
Each run's outcome goes to the store's IJobOutcomes, :wagoe/metrics,
:wagoe/error-reporting and :on-outcome (wagoe.jobs.shell.outcomes).
:wagoe/job-tenant-runner With the tenant module on: what runs a job that carries a tenant in that tenant's schema. The workers take it; without it such a job fails.
:wagoe/job-dead-letter-retention
The :jobs/dead-letter retention policy, contributed under :retention.
Runs only when :wagoe/privacy :retention names it.
Until BOU-418 this namespace wired none of it: :wagoe/jobs was a settings
passthrough, so java -jar wagoe.jar worker booted and processed nothing.
Integrant wiring for the jobs module.
Config keys:
:wagoe/jobs
The settings block that switches the module on:
{:provider :memory | :db | :redis ; default :memory
:lease-ms 60000 ; :db only, in-flight lease
:redis {:host "localhost" :port 6379}
:workers {:count 1 :queue-name :default :virtual-threads? false}
:recurring {:nightly-report {:cron "0 2 * * *" :zone "Europe/Amsterdam"}}
:scheduler {:tick-ms 5000 :lease-ms 30000 :zone "UTC"}
:on-outcome my.app.jobs/on-outcome ; optional, (fn [outcome])
:report-failures-every-ms 3600000 ; error-tracker throttle
:expect-success {:nightly-report 26}} ; job type -> hours, else reported
:wagoe/jobs-runtime
Queue, store and stats from one adapter, so the three share a backend.
:wagoe/job-queue, :wagoe/job-store, :wagoe/job-stats
What other modules and the devtools dashboard take a ref to — `workflow`
and `push` already document `:job-queue`. Projections of the runtime rather
than components of their own, because the in-memory adapter's queue and
store must share state. `:wagoe/job-stats` is nil under `:provider :db`,
which ships no IJobStats.
:wagoe/idempotency-store
`IIdempotencyStore` on the runtime's backend, for any module.
:wagoe/job-scheduler
The recurring-job ticker: `:recurring {job-type {:cron "0 2 * * *" …}}`
from the settings block, plus whatever `recurring/schedule-recurring!`
adds. `:scheduler {:tick-ms :lease-ms :zone :watch-ms}` tunes it. With
`:expect-success` its lease holder reports a job type that has not
succeeded within its hours (`outcomes/check-stale!`).
:wagoe/job-registry
The handler map every module contributed. Modules contribute handlers the
way they contribute routes: a `:job-handlers` vector in their `ig-config`,
collected by `wagoe.platform.shell.system.config` (BOU-330 pattern).
:wagoe/job-workers
The worker pool. `:count 0` builds none — that is a web node in the
web/worker split; the default of 1 is a single process that both enqueues
and runs, which is what an application without a deployment topology has.
Each run's outcome goes to the store's `IJobOutcomes`, `:wagoe/metrics`,
`:wagoe/error-reporting` and `:on-outcome` (`wagoe.jobs.shell.outcomes`).
:wagoe/job-tenant-runner
With the tenant module on: what runs a job that carries a tenant in that
tenant's schema. The workers take it; without it such a job fails.
:wagoe/job-dead-letter-retention
The `:jobs/dead-letter` retention policy, contributed under `:retention`.
Runs only when `:wagoe/privacy :retention` names it.
Until BOU-418 this namespace wired none of it: `:wagoe/jobs` was a settings
passthrough, so `java -jar wagoe.jar worker` booted and processed nothing.Where the worker reports how each run ended (BOU-832): the store's record, metrics, the error tracker and the application's own hook. Each is optional, and none can fail a job.
Failure and staleness reports share one throttle per job type, a claim in the idempotency store, so nodes sharing that store report once between them (BOU-834).
Where the worker reports how each run ended (BOU-832): the store's record, metrics, the error tracker and the application's own hook. Each is optional, and none can fail a job. Failure and staleness reports share one throttle per job type, a claim in the idempotency store, so nodes sharing that store report once between them (BOU-834).
Recurring jobs on cron schedules, and enqueue-once by idempotency key.
Every replica may run a ticker; the one holding the lease in
IRecurringState enqueues. Each tick is claimed in the idempotency store
under (job type, scheduled instant) before it is enqueued, so two tickers
that overlap after a lease handover still enqueue it once.
Ticks missed while no ticker ran collapse into one: after downtime a schedule fires its latest missed tick once and the earlier ones never. A schedule's first tick is the first after it was registered.
The lease holder warns once about a scheduled job type with no handler here. Not refused at init: handlers may be registered after boot, or live only on worker nodes.
Recurring jobs on cron schedules, and enqueue-once by idempotency key. Every replica may run a ticker; the one holding the lease in `IRecurringState` enqueues. Each tick is claimed in the idempotency store under (job type, scheduled instant) before it is enqueued, so two tickers that overlap after a lease handover still enqueue it once. Ticks missed while no ticker ran collapse into one: after downtime a schedule fires its latest missed tick once and the earlier ones never. A schedule's first tick is the first after it was registered. The lease holder warns once about a scheduled job type with no handler here. Not refused at init: handlers may be registered after boot, or live only on worker nodes.
Tenant context support for background job processing.
This namespace provides utilities for executing background jobs within tenant-specific database schema contexts. It integrates with the multi-tenant architecture by extracting tenant-id from job metadata and setting the appropriate PostgreSQL search_path before job execution.
Key Features:
Usage: (require '[wagoe.jobs.shell.tenant-context :as tenant-jobs])
;; Enqueue job with tenant context (tenant-jobs/enqueue-tenant-job! job-queue tenant-id :send-email {:to "user@example.com"})
;; Process job with tenant context (tenant-jobs/process-tenant-job! job handler-fn db-ctx tenant-service)
See ADR-004 for architecture details (lines 525-554).
Tenant context support for background job processing.
This namespace provides utilities for executing background jobs within
tenant-specific database schema contexts. It integrates with the multi-tenant
architecture by extracting tenant-id from job metadata and setting the
appropriate PostgreSQL search_path before job execution.
Key Features:
- Extract tenant-id from job metadata
- Set database search_path to tenant schema
- Execute job handlers in tenant context
- Automatic fallback to public schema if no tenant
Usage:
(require '[wagoe.jobs.shell.tenant-context :as tenant-jobs])
;; Enqueue job with tenant context
(tenant-jobs/enqueue-tenant-job! job-queue tenant-id :send-email
{:to "user@example.com"})
;; Process job with tenant context
(tenant-jobs/process-tenant-job! job handler-fn db-ctx tenant-service)
See ADR-004 for architecture details (lines 525-554).Background job worker implementation.
Workers poll job queues, execute jobs, and handle retries. This module provides a production-ready worker with:
Background job worker implementation. Workers poll job queues, execute jobs, and handle retries. This module provides a production-ready worker with: - Graceful shutdown - Configurable polling intervals - Automatic retry handling - Scheduled job processing - Job handler registry - Comprehensive error handling
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |