signal.data-platform-telemetry.metrics — com.signal-ai/data-platform-telemetry 0.4.10

*default-registry* clj

create-registry clj

Create a fresh iapetos collector registry.

decrement-messages-in-flight clj

(decrement-messages-in-flight registry)

(decrement-messages-in-flight registry n)

Decrement the number of messages currently being processed by the service.

document-latency-histogram-buckets clj

(document-latency-histogram-buckets)

(document-latency-histogram-buckets {:keys [target-slo low-resolution-step
                                            high-resolution-step
                                            max-bucket-value exponential-factor]
                                     :or {target-slo 90
                                          low-resolution-step 15
                                          high-resolution-step 5
                                          max-bucket-value (* 180 60)
                                          exponential-factor 1.2}})

Generate buckets for document latency histograms. Works for both pipeline and queue-worker. Defaults are provided as an example for pipeline latency.

This function should be used in the REPL to generate latency histogram buckets

exponential-buckets clj

(exponential-buckets start factor max-value)

(exponential-buckets start factor max-value initial-step)

Generates buckets with exponentially growing sizes based on <factor>, with initial step <initial-step>, starting at <start>, and ending with the bucket including the <max-value> - this may be larger than the max-value to include it.

If initial step is not provided, <start> is used as initial step.

get-default-histogram-buckets clj

(get-default-histogram-buckets {:keys [max-batch-size] :or {max-batch-size 50}})

Returns the default buckets for histograms for the metrics library.

Buckets can subsequently be overwritten by overwriting a given key.

increment-messages-in-flight clj

(increment-messages-in-flight registry)

(increment-messages-in-flight registry n)

Increment the number of messages currently being processed by the service.

init clj

(init)

(init {:keys [registry ring? jvm?]
       :or {ring? true jvm? true registry *default-registry*}})

Initialises a Prometheus metrics registry and returns it.

message-ack-time clj macro

(message-ack-time registry & body)

Track the total time taken by the macro body to acknowledge (i.e. output) the message result, e.g. writing to S3 and publishing to SNS.

message-process-time clj macro

(message-process-time registry & body)

Track the time taken by the macro body to run the service logic on the message.

message-publish-time clj macro

(message-publish-time registry & body)

Track the time taken by the macro body to publish a new onward message, e.g to SNS.

message-read-time clj macro

(message-read-time registry & body)

Track the time taken by the macro body to read data for the message from storage, e.g S3.

message-result-write-time clj macro

(message-result-write-time registry & body)

Track the time taken by the macro body to write the message result to storage, e.g S3.

message-total-time clj macro

(message-total-time registry & body)

Track the total time taken by the macro body to run the read the message, run logic, and acknoweledge it.

observe-message-ack-time clj

Track the total time taken to acknowledge (i.e. output) the message result, e.g. writing to S3 and publishing to SNS.

observe-message-batch-size clj

(observe-message-batch-size registry size)

Observe the size of a batch being processed, e.g. 50 for an article mini-batch, or 1 for a single document.

observe-message-process-time clj

Track the time taken to run the service logic on the message.

observe-message-publish-time clj

Track the time taken to publish a new onward message, e.g to SNS.

observe-message-read-time clj

Track the time taken to read data for the message from storage, e.g S3.

observe-message-received-times clj

(observe-message-received-times registry times-count)

Track the number of times the message was received (> 1 means this is a retry)

observe-message-result-write-time clj

Track the time taken to write the message result to storage, e.g S3.

observe-message-total-time clj

Track the total time taken to run the read the message, run logic, and acknoweledge it.

observe-queue-worker-error clj

(observe-queue-worker-error registry _)

Observe an error in processing.

observe-time-seconds clj

(observe-time-seconds metric registry start-timestamp-ms end-timestamp-ms)

Given a metric name and a start/end timestamp in milliseconds, observes the time in seconds between the two times.

observe-total-document-latency clj

Track the total time (seconds) between when upstream service sent message to the queue and this service sent message to output topic after processing it

pipeline-latency clj

(pipeline-latency registry {:keys [histogram-bucket-overrides] :as config})

Registers metrics to track latency from point-to-point in a pipeline to the given registry.

The configuration histogram-bucket-overrides.pipeline-document-latency must be provided - be sure to include the explicit value of any SLOs in your bucket distribution.

queue-reader clj

(queue-reader registry {:keys [histogram-bucket-overrides] :as config})

Registers queue reader metrics on the given registry.

queue-writer clj

(queue-writer registry {:keys [histogram-bucket-overrides] :as config})

Registers queue worker metrics on the given registry.

record-pipeline-latency clj

(record-pipeline-latency registry
                         {:keys [mode measured-from]}
                         document-received-date-time)

Observe pipeline latency metrics for a document.

These metrics are labelled by

• "mode": the mode of operation of the service if applicable, e.g. realtime, non-realtime
• "measured-from": where the latency was measured from, e.g. "ingestion" or "pipeline_start"

ring-wrap-instrumentation clj

(ring-wrap-instrumentation app registry)

Wraps a ring application in Prometheus instrumentation.

ring-wrap-metrics-expose clj

(ring-wrap-metrics-expose app registry)

Creates a ring handler which exposes Prometheus metrics for the given registry on /internal/metrics.

take-while-and-n-more clj

(take-while-and-n-more pred n coll)