Liking cljdoc? Tell your friends :D

wagoe.jobs.shell.adapters.codec

How the :db and :redis queues store a job: transit, so a handler gets back the keywords, UUIDs and instants it was given, as the in-memory queue does. They stored JSON until BOU-689, and jobs written then still decode.

How the :db and :redis queues store a job: transit, so a handler gets back
the keywords, UUIDs and instants it was given, as the in-memory queue does.
They stored JSON until BOU-689, and jobs written then still decode.
raw docstring

wagoe.jobs.shell.adapters.db

Database-backed job queue (IJobQueue) on next.jdbc.

Lets an app run durable background jobs on its existing SQL database (H2, PostgreSQL or MySQL) without standing up Redis, and survive a Redis outage. Implements the same reliable-dequeue contract as the Redis adapter:

  • dequeue-job! atomically claims the next ready row (sets status=processing, locked_by/locked_at) so a worker that crashes mid-job does not lose it.
  • ack-job! removes the row once the worker is done with it.
  • reclaim-abandoned-jobs! returns rows whose lease has expired (locked_at older than lease-ms) back to ready, i.e. at-least-once.

Portable SQL only (no SELECT ... FOR UPDATE SKIP LOCKED): the claim is a SELECT of the best candidate followed by a conditional UPDATE ... WHERE status='ready'. Concurrent workers that pick the same row have their UPDATEs serialized by the row lock — exactly one wins (update count 1); the losers retry the next candidate. This runs identically on H2, PostgreSQL and MySQL, so the reliability tests run on the default in-memory H2 test database.

The authoritative job value is the transit payload column (same wire format as the Redis adapter, wagoe.jobs.shell.adapters.codec). The indexed columns (queue, priority_rank, status, execute_at, locked_at) exist only for ordering, claiming, and reclaim.

The worker renews the lease while a handler runs (ports/ILeasedJobQueue), so lease-ms bounds how long a crashed worker's job waits, not how long a job may run (BOU-823).

Caveats:

  • payload is VARCHAR(1000000) (LONGTEXT on MySQL); keep serialized jobs under ~1 MB.
Database-backed job queue (IJobQueue) on next.jdbc.

Lets an app run durable background jobs on its existing SQL database (H2,
PostgreSQL or MySQL) without standing up Redis, and survive a Redis outage. Implements
the same reliable-dequeue contract as the Redis adapter:

- `dequeue-job!` atomically claims the next ready row (sets status=processing,
  locked_by/locked_at) so a worker that crashes mid-job does not lose it.
- `ack-job!` removes the row once the worker is done with it.
- `reclaim-abandoned-jobs!` returns rows whose lease has expired
  (locked_at older than `lease-ms`) back to `ready`, i.e. at-least-once.

Portable SQL only (no `SELECT ... FOR UPDATE SKIP LOCKED`): the claim is a
SELECT of the best candidate followed by a conditional `UPDATE ... WHERE
status='ready'`. Concurrent workers that pick the same row have their UPDATEs
serialized by the row lock — exactly one wins (update count 1); the losers
retry the next candidate. This runs identically on H2, PostgreSQL and MySQL, so the
reliability tests run on the default in-memory H2 test database.

The authoritative job value is the transit `payload` column (same wire format
as the Redis adapter, `wagoe.jobs.shell.adapters.codec`). The
indexed columns (queue, priority_rank, status, execute_at, locked_at) exist
only for ordering, claiming, and reclaim.

The worker renews the lease while a handler runs (`ports/ILeasedJobQueue`),
so `lease-ms` bounds how long a crashed worker's job waits, not how long a
job may run (BOU-823).

Caveats:
- `payload` is `VARCHAR(1000000)` (`LONGTEXT` on MySQL); keep serialized
  jobs under ~1 MB.
raw docstring

wagoe.jobs.shell.adapters.in-memory

In-memory job queue implementation for development and testing.

Uses Clojure atoms and data structures for job queuing. This adapter is suitable for:

  • Local development without Redis
  • Fast unit testing
  • CI/CD pipelines
  • Learning and tutorials

NOT suitable for:

  • Production use (no persistence, single-process only)
  • Distributed systems (not shared across processes)
  • High-volume job processing (limited by memory)

State is stored in atoms with the following structure:

  • jobs: Map of job-id -> job
  • queues: Map of queue-name -> priority-map of [priority job-ids]
  • scheduled: Sorted set of [execute-at job-id] pairs
  • failed: Vector of failed job-ids
  • stats: Map of queue-name -> statistics
In-memory job queue implementation for development and testing.

Uses Clojure atoms and data structures for job queuing. This adapter is
suitable for:
- Local development without Redis
- Fast unit testing
- CI/CD pipelines
- Learning and tutorials

NOT suitable for:
- Production use (no persistence, single-process only)
- Distributed systems (not shared across processes)
- High-volume job processing (limited by memory)

State is stored in atoms with the following structure:
- jobs: Map of job-id -> job
- queues: Map of queue-name -> priority-map of [priority job-ids]
- scheduled: Sorted set of [execute-at job-id] pairs
- failed: Vector of failed job-ids
- stats: Map of queue-name -> statistics
raw docstring

wagoe.jobs.shell.adapters.redis

Redis-backed job queue implementation.

Uses Redis for distributed job queuing with the following Redis data structures:

  • Sorted Sets: For scheduled jobs (scored by execute-at timestamp)
  • Lists: For priority queues (critical, high, normal, low)
  • Hashes: For job data storage
  • Sets: For tracking workers

This adapter provides production-grade job queuing with:

  • Distributed queue across multiple workers
  • Priority-based job processing
  • Scheduled job execution
  • Job persistence
  • Atomic operations
Redis-backed job queue implementation.

Uses Redis for distributed job queuing with the following Redis data structures:
- Sorted Sets: For scheduled jobs (scored by execute-at timestamp)
- Lists: For priority queues (critical, high, normal, low)
- Hashes: For job data storage
- Sets: For tracking workers

This adapter provides production-grade job queuing with:
- Distributed queue across multiple workers
- Priority-based job processing
- Scheduled job execution
- Job persistence
- Atomic operations
raw docstring

wagoe.jobs.shell.module-wiring

Integrant wiring for the jobs module.

Config keys:

:wagoe/jobs The settings block that switches the module on: {:provider :memory | :db | :redis ; default :memory :lease-ms 60000 ; :db only, in-flight lease :redis {:host "localhost" :port 6379} :workers {:count 1 :queue-name :default :virtual-threads? false} :recurring {:nightly-report {:cron "0 2 * * *" :zone "Europe/Amsterdam"}} :scheduler {:tick-ms 5000 :lease-ms 30000 :zone "UTC"} :on-outcome my.app.jobs/on-outcome ; optional, (fn [outcome]) :report-failures-every-ms 3600000 ; error-tracker throttle :expect-success {:nightly-report 26}} ; job type -> hours, else reported

:wagoe/jobs-runtime Queue, store and stats from one adapter, so the three share a backend.

:wagoe/job-queue, :wagoe/job-store, :wagoe/job-stats What other modules and the devtools dashboard take a ref to — workflow and push already document :job-queue. Projections of the runtime rather than components of their own, because the in-memory adapter's queue and store must share state. :wagoe/job-stats is nil under :provider :db, which ships no IJobStats.

:wagoe/idempotency-store IIdempotencyStore on the runtime's backend, for any module.

:wagoe/job-scheduler The recurring-job ticker: :recurring {job-type {:cron "0 2 * * *" …}} from the settings block, plus whatever recurring/schedule-recurring! adds. :scheduler {:tick-ms :lease-ms :zone :watch-ms} tunes it. With :expect-success its lease holder reports a job type that has not succeeded within its hours (outcomes/check-stale!).

:wagoe/job-registry The handler map every module contributed. Modules contribute handlers the way they contribute routes: a :job-handlers vector in their ig-config, collected by wagoe.platform.shell.system.config (BOU-330 pattern).

:wagoe/job-workers The worker pool. :count 0 builds none — that is a web node in the web/worker split; the default of 1 is a single process that both enqueues and runs, which is what an application without a deployment topology has. Each run's outcome goes to the store's IJobOutcomes, :wagoe/metrics, :wagoe/error-reporting and :on-outcome (wagoe.jobs.shell.outcomes).

:wagoe/job-tenant-runner With the tenant module on: what runs a job that carries a tenant in that tenant's schema. The workers take it; without it such a job fails.

:wagoe/job-dead-letter-retention The :jobs/dead-letter retention policy, contributed under :retention. Runs only when :wagoe/privacy :retention names it.

Until BOU-418 this namespace wired none of it: :wagoe/jobs was a settings passthrough, so java -jar wagoe.jar worker booted and processed nothing.

Integrant wiring for the jobs module.

Config keys:

:wagoe/jobs
  The settings block that switches the module on:
    {:provider :memory | :db | :redis     ; default :memory
     :lease-ms 60000                      ; :db only, in-flight lease
     :redis    {:host "localhost" :port 6379}
     :workers  {:count 1 :queue-name :default :virtual-threads? false}
     :recurring {:nightly-report {:cron "0 2 * * *" :zone "Europe/Amsterdam"}}
     :scheduler {:tick-ms 5000 :lease-ms 30000 :zone "UTC"}
     :on-outcome my.app.jobs/on-outcome   ; optional, (fn [outcome])
     :report-failures-every-ms 3600000    ; error-tracker throttle
     :expect-success {:nightly-report 26}} ; job type -> hours, else reported

:wagoe/jobs-runtime
  Queue, store and stats from one adapter, so the three share a backend.

:wagoe/job-queue, :wagoe/job-store, :wagoe/job-stats
  What other modules and the devtools dashboard take a ref to — `workflow`
  and `push` already document `:job-queue`. Projections of the runtime rather
  than components of their own, because the in-memory adapter's queue and
  store must share state. `:wagoe/job-stats` is nil under `:provider :db`,
  which ships no IJobStats.

:wagoe/idempotency-store
  `IIdempotencyStore` on the runtime's backend, for any module.

:wagoe/job-scheduler
  The recurring-job ticker: `:recurring {job-type {:cron "0 2 * * *" …}}`
  from the settings block, plus whatever `recurring/schedule-recurring!`
  adds. `:scheduler {:tick-ms :lease-ms :zone :watch-ms}` tunes it. With
  `:expect-success` its lease holder reports a job type that has not
  succeeded within its hours (`outcomes/check-stale!`).

:wagoe/job-registry
  The handler map every module contributed. Modules contribute handlers the
  way they contribute routes: a `:job-handlers` vector in their `ig-config`,
  collected by `wagoe.platform.shell.system.config` (BOU-330 pattern).

:wagoe/job-workers
  The worker pool. `:count 0` builds none — that is a web node in the
  web/worker split; the default of 1 is a single process that both enqueues
  and runs, which is what an application without a deployment topology has.
  Each run's outcome goes to the store's `IJobOutcomes`, `:wagoe/metrics`,
  `:wagoe/error-reporting` and `:on-outcome` (`wagoe.jobs.shell.outcomes`).

:wagoe/job-tenant-runner
  With the tenant module on: what runs a job that carries a tenant in that
  tenant's schema. The workers take it; without it such a job fails.

:wagoe/job-dead-letter-retention
  The `:jobs/dead-letter` retention policy, contributed under `:retention`.
  Runs only when `:wagoe/privacy :retention` names it.

Until BOU-418 this namespace wired none of it: `:wagoe/jobs` was a settings
passthrough, so `java -jar wagoe.jar worker` booted and processed nothing.
raw docstring

wagoe.jobs.shell.outcomes

Where the worker reports how each run ended (BOU-832): the store's record, metrics, the error tracker and the application's own hook. Each is optional, and none can fail a job.

Failure and staleness reports share one throttle per job type, a claim in the idempotency store, so nodes sharing that store report once between them (BOU-834).

Where the worker reports how each run ended (BOU-832): the store's record,
metrics, the error tracker and the application's own hook. Each is
optional, and none can fail a job.

Failure and staleness reports share one throttle per job type, a claim in
the idempotency store, so nodes sharing that store report once between
them (BOU-834).
raw docstring

wagoe.jobs.shell.recurring

Recurring jobs on cron schedules, and enqueue-once by idempotency key.

Every replica may run a ticker; the one holding the lease in IRecurringState enqueues. Each tick is claimed in the idempotency store under (job type, scheduled instant) before it is enqueued, so two tickers that overlap after a lease handover still enqueue it once.

Ticks missed while no ticker ran collapse into one: after downtime a schedule fires its latest missed tick once and the earlier ones never. A schedule's first tick is the first after it was registered.

The lease holder warns once about a scheduled job type with no handler here. Not refused at init: handlers may be registered after boot, or live only on worker nodes.

Recurring jobs on cron schedules, and enqueue-once by idempotency key.

Every replica may run a ticker; the one holding the lease in
`IRecurringState` enqueues. Each tick is claimed in the idempotency store
under (job type, scheduled instant) before it is enqueued, so two tickers
that overlap after a lease handover still enqueue it once.

Ticks missed while no ticker ran collapse into one: after downtime a
schedule fires its latest missed tick once and the earlier ones never. A
schedule's first tick is the first after it was registered.

The lease holder warns once about a scheduled job type with no handler
here. Not refused at init: handlers may be registered after boot, or live
only on worker nodes.
raw docstring

wagoe.jobs.shell.tenant-context

Tenant context support for background job processing.

This namespace provides utilities for executing background jobs within tenant-specific database schema contexts. It integrates with the multi-tenant architecture by extracting tenant-id from job metadata and setting the appropriate PostgreSQL search_path before job execution.

Key Features:

  • Extract tenant-id from job metadata
  • Set database search_path to tenant schema
  • Execute job handlers in tenant context
  • Automatic fallback to public schema if no tenant

Usage: (require '[wagoe.jobs.shell.tenant-context :as tenant-jobs])

;; Enqueue job with tenant context (tenant-jobs/enqueue-tenant-job! job-queue tenant-id :send-email {:to "user@example.com"})

;; Process job with tenant context (tenant-jobs/process-tenant-job! job handler-fn db-ctx tenant-service)

See ADR-004 for architecture details (lines 525-554).

Tenant context support for background job processing.

This namespace provides utilities for executing background jobs within
tenant-specific database schema contexts. It integrates with the multi-tenant
architecture by extracting tenant-id from job metadata and setting the
appropriate PostgreSQL search_path before job execution.

Key Features:
- Extract tenant-id from job metadata
- Set database search_path to tenant schema
- Execute job handlers in tenant context
- Automatic fallback to public schema if no tenant

Usage:
  (require '[wagoe.jobs.shell.tenant-context :as tenant-jobs])
  
  ;; Enqueue job with tenant context
  (tenant-jobs/enqueue-tenant-job! job-queue tenant-id :send-email
                                   {:to "user@example.com"})
  
  ;; Process job with tenant context
  (tenant-jobs/process-tenant-job! job handler-fn db-ctx tenant-service)

See ADR-004 for architecture details (lines 525-554).
raw docstring

wagoe.jobs.shell.worker

Background job worker implementation.

Workers poll job queues, execute jobs, and handle retries. This module provides a production-ready worker with:

  • Graceful shutdown
  • Configurable polling intervals
  • Automatic retry handling
  • Scheduled job processing
  • Job handler registry
  • Comprehensive error handling
Background job worker implementation.

Workers poll job queues, execute jobs, and handle retries. This module
provides a production-ready worker with:
- Graceful shutdown
- Configurable polling intervals
- Automatic retry handling
- Scheduled job processing
- Job handler registry
- Comprehensive error handling
raw docstring

cljdoc builds & hosts documentation for Clojure/Script libraries

Keyboard shortcuts
Ctrl+kJump to recent docs
←Move to previous article
→Move to next article
Ctrl+/Jump to the search field
× close