The speech models that this process holds in memory (#448).
A loader reports each model that it loads with loaded!: its size, a function
that tells if a request uses it, and a function that releases it. A request that
uses a model calls used!. One background thread then:
speech_model_idle_unload), andspeech_model_memory).It never releases a model that a request uses, and it never releases the most
recently used model for the limit, so one model that is larger than the limit
still works. The next request loads a released model again. /healthz shows
the list from status.
The speech models that this process holds in memory (#448).
A loader reports each model that it loads with `loaded!`: its size, a function
that tells if a request uses it, and a function that releases it. A request that
uses a model calls `used!`. One background thread then:
- releases a model that nothing used for the idle time
(setting `speech_model_idle_unload`), and
- releases the least recently used models while the loaded models need more
than the memory limit (setting `speech_model_memory`).
It never releases a model that a request uses, and it never releases the most
recently used model for the limit, so one model that is larger than the limit
still works. The next request loads a released model again. `/healthz` shows
the list from `status`.The registry that this thread reads and writes. Tests bind a fresh atom.
The registry that this thread reads and writes. Tests bind a fresh atom.
(choice-bytes choice)Bytes of a speech_model_memory choice such as "4 GB". nil means no limit.
Bytes of a `speech_model_memory` choice such as "4 GB". nil means no limit.
(choice-ms choice)Milliseconds of a speech_model_idle_unload choice such as "5 minutes". nil
means never.
Milliseconds of a `speech_model_idle_unload` choice such as "5 minutes". nil means never.
(file-bytes paths)The total size of the regular files in paths. A directory counts all of its
files. A missing path counts zero.
The total size of the regular files in `paths`. A directory counts all of its files. A missing path counts zero.
(loaded! id entry)Record that model id is in memory now. entry has:
:token the loaded object; released! compares it
:bytes the size of the model
:release! frees the model; returns false when a request uses it
:in-use? optional; true while a request uses the model
:keep-toggle optional; a boolean setting that keeps the model when it is idle
The background thread applies the memory limit soon after.
Record that model `id` is in memory now. `entry` has: :token the loaded object; `released!` compares it :bytes the size of the model :release! frees the model; returns false when a request uses it :in-use? optional; true while a request uses the model :keep-toggle optional; a boolean setting that keeps the model when it is idle The background thread applies the memory limit soon after.
(released! id token)Remove model id when token is still the loaded object.
Remove model `id` when `token` is still the loaded object.
(status)The loaded speech models for /healthz, most recently used first.
The loaded speech models for `/healthz`, most recently used first.
(sweep!)(sweep! opts)Release the models that victims names. Returns the released ids. opts may
give :now, :limit and :idle-ms; the settings give the missing ones. A model
that a request started to use after the check stays.
Release the models that `victims` names. Returns the released ids. `opts` may give `:now`, `:limit` and `:idle-ms`; the settings give the missing ones. A model that a request started to use after the check stays.
How often the background thread looks for idle models.
How often the background thread looks for idle models.
(used! id)Record that a request uses model id now.
Record that a request uses model `id` now.
(victims models now limit idle-ms)The models to release now, as [entry reason]. First the idle models, then the
least recently used models while the rest need more than limit bytes. Never a
model in use, and never the most recently used model for the limit. limit or
idle-ms nil turns that rule off.
The models to release now, as `[entry reason]`. First the idle models, then the least recently used models while the rest need more than `limit` bytes. Never a model in use, and never the most recently used model for the limit. `limit` or `idle-ms` nil turns that rule off.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |