verbo.sokka.supervisor — net.clojars.sathyavijayan/sokka 0.0.1-SNAPSHOT.240521.1534

Monitors running and snoozed tasks and revokes the lease of tasks whose lease has expired. Locking implementations in distributed systems are prone to issues caused by clock skew. A process with skewed time can cause the active and valid lease held by other processes to be revoked. This can cause thrashing of tasks and results in the same task being attempted to run multiple times wasting system resources and delaying sucessful task execution. This issue can be resolved by maintaining an in memory list of tasks with active lease and a projected expiry time based on the current time of the process/instance. If the task still has the same record version after the expiry, it is considered stale and its lease is forcefully revoked freeing it up to be reserved again. The projected expiry time calculated based on the current time (for running tasks, it is current time + lease time, and for snoozed tasks it is current time + snooze time). The cleanup-leased-tasks! function implements a mark-sweep pattern, where the state of leased tasks are updated in an agent during the 'mark' stage and the expired tasks are revoked in the 'sweep' state.

cleanup-leased-tasks! clj

(cleanup-leased-tasks! monitored-tasks
                       taskq
                       topic
                       {:keys [lease-time-ms] :as opts})

mark-task! clj

(mark-task! m
            taskq
            {:keys [lease-time-ms] :as opts}
            {:keys [task-id record-ver status snooze-time] :as task})

'marks' the last seen record version and expiry time for the given task. expiry is computed by adding lease-time-ms to the current time for running tasks and snooze-time for snoozed tasks. If there is not entry for the given task-id, a new entry is added to m. If an entry already exists, and if the record version in the entry is not the same as the task the entry is updated, otherwise, the entry is left as-is.

sweep-task! clj

(sweep-task! a taskq task-id record-ver)

Revoke lease of a given task, so it will be available for reservation. The record-ver of the task is passed to the update function to ensure that tasks which were updated concurrently else-where are not overwritten by mistake. Any exceptions thrown during the update except throttling-exception are ignored and the record for the task is removed from the map m.