event-data-common.checkpoint — event-data-common 0.1.57

Checkpoint functionality for Agents. Allows setting of checkpoints and function to check if the checkpoint happened more than a period of time ago. Checkpoint identifiers are anything that can be converted to a string, and are hashed.

This is used so that Agents can run in continual loops, only re-trying when a given time period has elapsed, choosing which items to checkpoint on (e.g. per whole domain scan, or per newsfeed).

Doing it this way makes it possible to resume a scan in the face of a crash. It also means that we can add new domains in future, and the Reddit source could back-scan for that domain, regardless of others.

Checkpoints are stored as JSON in case we want to inspect them later. Longs stored as Strings because you can't really trust numbers in JSON.

checkpoint-store clj

floor-date clj

(floor-date date period-ago)

If the supplied date is nil or earlier than the time-period, return the time-period-ago date.

get-checkpoint clj

(get-checkpoint identifier)

For a given checkpoint, return the date time that it happened, or nil

get-floored-checkpoint-date clj

(get-floored-checkpoint-date identifier period-ago)

Get the checkpoint date, limited to maximum period-ago.

has-time-elapsed? clj

(has-time-elapsed? identifier period-ago)

Has at least the given time elapsed since the last checkpoint for this identifier? If so, return the last date. If not, return nil.

hash-identifier clj

(hash-identifier identifier)

run-checkpointed! clj

(run-checkpointed! identifier since-period-ago floor-period-ago function)

Run the given function and set checkpoint with identifier if there hasn't been a checkpoint in the given time period. Call function with date of last run, or floor-period-ago if it never ran before

run-once-checkpointed! clj

(run-once-checkpointed! identifier function)

Run the supplied function with checkpoint identifier only once, ever.

set-checkpoint! clj

(set-checkpoint! identifier)

(set-checkpoint! identifier time-value)

Set checkpoint with identifier to given time, or now.