Prompt-cache keepalive for one session. After a real provider request, it plans
a warm request shortly before the provider cache TTL ends. A long tool run, or a
short pause in idle mode, then keeps the cache. A warm runs only when its
expected saving beats its cost.
Prompt-cache keepalive for one session. After a real provider request, it plans a warm request shortly before the provider cache TTL ends. A long tool run, or a short pause in `idle` mode, then keeps the cache. A warm runs only when its expected saving beats its cost.
(create)(create {:keys [clock schedule]})Returns the prompt-cache warmer of one session. Tests can pass :clock, a
function that returns epoch milliseconds, and :schedule, a function of a delay
and a task that returns a cancel function.
Returns the prompt-cache warmer of one session. Tests can pass `:clock`, a function that returns epoch milliseconds, and `:schedule`, a function of a delay and a task that returns a cancel function.
(drain-costs! warmer accounting-atom)Adds the costs of finished warms to a turn's accounting-atom.
Adds the costs of finished warms to a turn's `accounting-atom`.
(late-limit-ms ttl-ms delay-ms)Returns how late a timer can fire and still warm: half of the time between the planned warm and the cache expiry. A later timer, for example after the computer slept, finds the cache gone.
Returns how late a timer can fire and still warm: half of the time between the planned warm and the cache expiry. A later timer, for example after the computer slept, finds the cache gone.
(request-finished! warmer {:keys [mode ttl-ms] :as request})Plans warms after a successful real request. request holds:
:mode - function that returns the prompt_cache_keepalive value
:ttl-ms - cache TTL of the serving route, or nil for no warm
:prompt-tokens - input tokens of the request, cached ones included
:pricing :provider :model - price the warm, see warm-savings-usd
:warm! - function of a cancel predicate that sends the warm
Plans warms after a successful real request. `request` holds: :mode - function that returns the `prompt_cache_keepalive` value :ttl-ms - cache TTL of the serving route, or nil for no warm :prompt-tokens - input tokens of the request, cached ones included :pricing :provider :model - price the warm, see `warm-savings-usd` :warm! - function of a cancel predicate that sends the warm
(request-started! warmer)Stops a planned or running warm, because a real provider request starts.
Stops a planned or running warm, because a real provider request starts.
(stop! warmer)Stops all warms of a closing session.
Stops all warms of a closing session.
(turn-settled! warmer)Ends the running phase when a turn ends. In idle mode, warms continue for 30
minutes with a lower reuse chance. In other modes, they stop.
Ends the running phase when a turn ends. In `idle` mode, warms continue for 30 minutes with a lower reuse chance. In other modes, they stop.
(warm-delay-ms ttl-ms)Returns the wait from a request to its warm in milliseconds: 90% of ttl-ms,
but at least 10 seconds before the cache expires. Returns nil when ttl-ms is
nil or leaves no room.
Returns the wait from a request to its warm in milliseconds: 90% of `ttl-ms`, but at least 10 seconds before the cache expires. Returns nil when `ttl-ms` is nil or leaves no room.
(warm-savings-usd {:keys [pricing model provider prompt-tokens]}
reuse-probability)Returns the expected saving of one warm in USD, or nil for an unpriced route. The
warm reads :prompt-tokens from the cache and writes one output token. Without
it, the next request writes the prompt to the cache again, with probability
reuse-probability. :pricing, :model and :provider price the tokens.
Returns the expected saving of one warm in USD, or nil for an unpriced route. The warm reads `:prompt-tokens` from the cache and writes one output token. Without it, the next request writes the prompt to the cache again, with probability `reuse-probability`. `:pricing`, `:model` and `:provider` price the tokens.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |