Borrowed EARS: a recording somebody attached, as text.
No provider wire carries audio. A voice memo dropped into a message is therefore
stored, played back for the human, and NAMED to the model — which is the same
dead end a blind model meets in front of a screenshot, and the reason
vision-describe exists. The answer here is the same shape, with one difference
that decides everything about where it runs: the transcript is not a second-hand
report bought from another provider, it is the recording's own WORDS, produced
locally by the speech engine this build already carries
(com.blockether.vis.internal.speech.core, normally Parakeet on this machine). It costs
no quota, it leaves no bytes on anybody's wire, and it is true of the file forever.
WHEN it runs is the whole design. A surface that STAGES a recording — the composer
rail, an upload, the gateway's own intake — calls request! the moment the file
arrives and paints outcome while the human is still typing. By the time the
turn is sent the words are normally already in hand; a turn that finds the work
still running JOINS it under a deadline instead of starting its own. Nothing is
ever transcribed twice, and nobody waits for a recording that was attached a
minute ago.
Four properties keep it affordable and honest:
pending,
unavailable, silent (statuses) — logged once and spelled on the wire as
transcription_status, because a blank band under a player must never be
indistinguishable from a recording that had no words in it. A failure is
remembered as a FAILURE and never as "no words"; a reason that describes this
moment rather than the file (a model still downloading, a toggle that is off) is
not remembered at all.A LEAF: attachments + voice + toggles, never back on the loop.
Borrowed EARS: a recording somebody attached, as text. No provider wire carries audio. A voice memo dropped into a message is therefore stored, played back for the human, and NAMED to the model — which is the same dead end a blind model meets in front of a screenshot, and the reason `vision-describe` exists. The answer here is the same shape, with one difference that decides everything about where it runs: the transcript is not a second-hand report bought from another provider, it is the recording's own WORDS, produced locally by the speech engine this build already carries (`com.blockether.vis.internal.speech.core`, normally Parakeet on this machine). It costs no quota, it leaves no bytes on anybody's wire, and it is true of the file forever. WHEN it runs is the whole design. A surface that STAGES a recording — the composer rail, an upload, the gateway's own intake — calls [[request!]] the moment the file arrives and paints [[outcome]] while the human is still typing. By the time the turn is sent the words are normally already in hand; a turn that finds the work still running JOINS it under a deadline instead of starting its own. Nothing is ever transcribed twice, and nobody waits for a recording that was attached a minute ago. Four properties keep it affordable and honest: - CONTENT-KEYED. Attachments replay on every later request of the session, so the digest of the bytes — not the position, not the filename — is the registry key, and a memo is transcribed exactly once per process. - ONE AT A TIME. Local speech saturates a core, so five memos in one message queue on one daemon worker rather than starting five decoders at once. - NEVER SILENT. Every miss is an OUTCOME carrying a reason — `pending`, `unavailable`, `silent` ([[statuses]]) — logged once and spelled on the wire as `transcription_status`, because a blank band under a player must never be indistinguishable from a recording that had no words in it. A failure is remembered as a FAILURE and never as "no words"; a reason that describes this moment rather than the file (a model still downloading, a toggle that is off) is not remembered at all. - TOTAL. No failure escapes: a recording that could not be transcribed is still stored, still played, and still named to the model. A LEAF: attachments + voice + toggles, never back on the loop.
User-message image, video, audio, document and gzip attachments.
Dropping a file onto the terminal pastes its PATH into the input (the terminal's drop behavior — same mechanism pi relies on). At turn start the engine scans the user message for path-shaped tokens that resolve to real image files, reads them, and attaches them to the initial user message as multimodal content blocks. Channel-neutral: every channel gets the same behavior because the scan runs in the engine, not the channel.
What a file IS is sniffed from magic bytes (pi-parity: jpeg / non-animated
png / gif / webp / bmp, plus MP4/QuickTime clips, mp3/m4a/wav/ogg/flac
recordings, PDF/HTML documents and gzip streams) or, for SVG, from the
markup head. Markdown uploads use a filename or MIME hint plus UTF-8 validation;
an extension alone never blesses a payload. A container the model cannot consume
is kept for the HUMAN and NAMED to the model (model-blind-media-type?).
Storing and SENDING are deliberately separate concerns:
max-image-bytes
is skipped with a reason the prompt assembler surfaces.wire-image is the one gate every image crosses on its way to
a provider: it decodes the payload to prove it is pixels, re-containers
what no wire accepts (BMP/SVG -> PNG via image-convert, on
com.blockether/imaging, so it behaves identically in the native image)
and REFUSES what it cannot turn into a picture.The split is what makes a bad attachment survivable. Attachments replay on every later turn, so a row blessed once on the way IN is shipped forever — one corrupt PNG that way is a permanent provider 400. Judged on the way OUT, the same row is simply dropped and the session keeps working.
User-message image, video, audio, document and gzip attachments.
Dropping a file onto the terminal pastes its PATH into the input (the
terminal's drop behavior — same mechanism pi relies on). At turn start
the engine scans the user message for path-shaped tokens that resolve
to real image files, reads them, and attaches them to the initial user
message as multimodal content blocks. Channel-neutral: every channel
gets the same behavior because the scan runs in the engine,
not the channel.
What a file IS is sniffed from magic bytes (pi-parity: jpeg / non-animated
png / gif / webp / bmp, plus MP4/QuickTime clips, mp3/m4a/wav/ogg/flac
recordings, PDF/HTML documents and gzip streams) or, for SVG, from the
markup head. Markdown uploads use a filename or MIME hint plus UTF-8 validation;
an extension alone never blesses a payload. A container the model cannot consume
is kept for the HUMAN and NAMED to the model ([[model-blind-media-type?]]).
Storing and SENDING are deliberately separate concerns:
* STORE — the original bytes under their sniffed container, nothing
converted, nothing downscaled, nothing re-compressed. What the user
supplied is what the session keeps, and a file over `max-image-bytes`
is skipped with a reason the prompt assembler surfaces.
* SEND — [[wire-image]] is the one gate every image crosses on its way to
a provider: it decodes the payload to prove it is pixels, re-containers
what no wire accepts (BMP/SVG -> PNG via `image-convert`, on
`com.blockether/imaging`, so it behaves identically in the native image)
and REFUSES what it cannot turn into a picture.
The split is what makes a bad attachment survivable. Attachments replay on
every later turn, so a row blessed once on the way IN is shipped forever —
one corrupt PNG that way is a permanent provider 400. Judged on the way OUT,
the same row is simply dropped and the session keeps working.Container conversion for attachment payloads -- NOT optimization.
vis stores and replays the attachment PICTURE verbatim: whatever the user
dropped is what is stored and what the provider sees, pixel for pixel (the
stored ENCODING may be shrunk losslessly -- compact). The one thing that
cannot be passed through is a container no vision wire accepts (attachments/ provider-image-media-types): a BMP is a perfectly good image and an SVG is
a perfectly good figure, and both are a hard 400 on every provider. Since an
attachment REPLAYS on every later turn, one such row kills the whole session.
So this namespace does exactly one thing: turn bytes a provider REFUSES into the same picture in a container it ACCEPTS.
.svg / gzipped .svgz) -> rendered PNG.mp4 / .mov) -> animated GIFEverything a renderer can answer is ASKED, not re-implemented.
com.blockether/imaging (Rust image + resvg over FFM) decodes, sniffs,
gunzips .svgz, resolves width="100%"/cm/viewBox letterboxing, and
rasterizes. It also owns the two BROWSER repairs resvg alone refuses, behind
imaging/svg-canvas:
viewBox.Both used to be hand-written Clojure HERE. They are Rust in the library now --
one implementation for every caller, and vis only asks for the answer.
renderer-delegation-test still pins that answer document by document, so a
change in resvg or in the repair shows up as a vis test failure.
Conversion is 1:1 and never re-compresses what the wire already takes: no
quality knob, no size heuristics on the way through. There are exactly two
exceptions, both narrow and both named: fit-within, which the send gate
calls ONLY for a payload over the wire's byte cap that would otherwise be
DROPPED, and compact, which the storage rail calls on the way into the
session_attachment BLOB and which is LOSSLESS -- same picture, fewer bytes,
verified by a re-probe. Nothing here ever throws: a
failure comes back as {:reason <why>} -- no :bytes, so the caller still
SKIPS the attachment, but it can TELL the user why instead of dropping a
perfectly valid picture in silence.
Pixels come from com.blockether/imaging, never from AWT/Java2D/ImageIO --
which is why this works identically in the native image, on every platform,
with no headless or fontconfig bootstrap. Tests do the opposite on purpose:
independent-decoder-test re-reads every payload this namespace emits with
javax.imageio, so "it decodes" is never just the encoder agreeing with
itself -- the provider's decoder is a third implementation again.
Container conversion for attachment payloads -- NOT optimization.
vis stores and replays the attachment PICTURE verbatim: whatever the user
dropped is what is stored and what the provider sees, pixel for pixel (the
stored ENCODING may be shrunk losslessly -- [[compact]]). The one thing that
cannot be passed through is a container no vision wire accepts (`attachments/
provider-image-media-types`): a BMP is a perfectly good image and an SVG is
a perfectly good figure, and both are a hard 400 on every provider. Since an
attachment REPLAYS on every later turn, one such row kills the whole session.
So this namespace does exactly one thing: turn bytes a provider REFUSES into
the same picture in a container it ACCEPTS.
* raster (BMP, TIFF, or anything else the decoder reads) -> PNG, 1:1
* vector (`.svg` / gzipped `.svgz`) -> rendered PNG
* video (`.mp4` / `.mov`) -> animated GIF
Everything a renderer can answer is ASKED, not re-implemented.
`com.blockether/imaging` (Rust `image` + resvg over FFM) decodes, sniffs,
gunzips `.svgz`, resolves `width="100%"`/`cm`/`viewBox` letterboxing, and
rasterizes. It also owns the two BROWSER repairs resvg alone refuses, behind
`imaging/svg-canvas`:
* a zero or negative declared size -- resvg refuses the document outright
("SVG has an invalid size"); browsers fall back to the `viewBox`.
* a document that declares NO size -- resvg's bounds run from the ORIGIN
(a figure at x=50 gains a 50px margin) and collapse to a bare 100x100
when content sits at negative coordinates, so the canvas is framed by the
ink actually painted.
Both used to be hand-written Clojure HERE. They are Rust in the library now --
one implementation for every caller, and vis only asks for the answer.
`renderer-delegation-test` still pins that answer document by document, so a
change in resvg or in the repair shows up as a vis test failure.
Conversion is 1:1 and never re-compresses what the wire already takes: no
quality knob, no size heuristics on the way through. There are exactly two
exceptions, both narrow and both named: [[fit-within]], which the send gate
calls ONLY for a payload over the wire's byte cap that would otherwise be
DROPPED, and [[compact]], which the storage rail calls on the way into the
`session_attachment` BLOB and which is LOSSLESS -- same picture, fewer bytes,
verified by a re-probe. Nothing here ever throws: a
failure comes back as `{:reason <why>}` -- no `:bytes`, so the caller still
SKIPS the attachment, but it can TELL the user why instead of dropping a
perfectly valid picture in silence.
Pixels come from `com.blockether/imaging`, never from AWT/Java2D/ImageIO --
which is why this works identically in the native image, on every platform,
with no headless or fontconfig bootstrap. Tests do the opposite on purpose:
`independent-decoder-test` re-reads every payload this namespace emits with
`javax.imageio`, so "it decodes" is never just the encoder agreeing with
itself -- the provider's decoder is a third implementation again.Snapshot deliberately linked workspace reports before persisting assistant prose.
Only the session's primary workspace is eligible, never other host roots. Hidden paths, credential-like names, policy exclusions, traversal, symlinks, directories and unsupported report types fail closed. Reads walk directory handles without following symlinks; platforms without SecureDirectoryStream refuse publication. At most eight distinct links and 8 MiB per file are delivered per iteration. Attachments are human-only snapshots, not live paths. Existing attachment and remote links, code examples and images are not republished. Failed local links become plain text with a recovery action; no host exception or file bytes leak.
Snapshot deliberately linked workspace reports before persisting assistant prose. Only the session's primary workspace is eligible, never other host roots. Hidden paths, credential-like names, policy exclusions, traversal, symlinks, directories and unsupported report types fail closed. Reads walk directory handles without following symlinks; platforms without SecureDirectoryStream refuse publication. At most eight distinct links and 8 MiB per file are delivered per iteration. Attachments are human-only snapshots, not live paths. Existing attachment and remote links, code examples and images are not republished. Failed local links become plain text with a recovery action; no host exception or file bytes leak.
Attachment storage-offload rail: a registry of storage BACKENDS plus the
pure OFFLOAD DECISION that routes one attachment's payload either INLINE
(bytes in the session_attachment.bytes BLOB) or EXTERNAL (bytes handed to
a backend, which returns a storage_uri -- scheme://... -- kept in the row
instead).
Zero SQL, zero schema change: the V4 session_attachment table already
carries a nullable storage_uri with an exactly-one(bytes, storage_uri)
CHECK. This namespace only decides WHICH of the two a given attachment takes,
PUTs/GETs the external bytes through the scheme-dispatched backend, and
hydrates a read-back envelope's :base64 from its :storage-uri on demand.
The decision is a PURE predicate hot? AND size (see default-offload?):
an image replays to a vision model every turn its iteration stays live, so it
is HOT -- kept inline even when large; a non-image artifact (PDF/CSV/wav/
download) is fetched at most once by a human, so it is COLD -- a good offload
candidate past a size floor. A backend may override the predicate wholesale
via :storage/offload?.
Precedence, engine-owned so the loop never learns a storage dialect:
:storage/offload? (the backend knows its own cost)default-offload? (engine default policy)Either way the bytes are COMPACTED first (image-convert/compact): a PNG /
JPEG / GIF payload is re-compressed LOSSLESSLY -- oxipng, jpegtran-style
marker stripping, gifsicle differencing -- before it becomes a BLOB or a
backend PUT. That is the only transformation this rail performs, it is
verified per payload (strictly smaller AND the same format at the same
dimensions and frame count, else the original is kept), and it is why a
matplotlib figure costs a fraction of a row.
Attachment storage-offload rail: a registry of storage BACKENDS plus the pure OFFLOAD DECISION that routes one attachment's payload either INLINE (bytes in the `session_attachment.bytes` BLOB) or EXTERNAL (bytes handed to a backend, which returns a `storage_uri` -- `scheme://...` -- kept in the row instead). Zero SQL, zero schema change: the V4 `session_attachment` table already carries a nullable `storage_uri` with an exactly-one(bytes, storage_uri) CHECK. This namespace only decides WHICH of the two a given attachment takes, PUTs/GETs the external bytes through the scheme-dispatched backend, and hydrates a read-back envelope's `:base64` from its `:storage-uri` on demand. The decision is a PURE predicate `hot? AND size` (see `default-offload?`): an image replays to a vision model every turn its iteration stays live, so it is HOT -- kept inline even when large; a non-image artifact (PDF/CSV/wav/ download) is fetched at most once by a human, so it is COLD -- a good offload candidate past a size floor. A backend may override the predicate wholesale via `:storage/offload?`. Precedence, engine-owned so the loop never learns a storage dialect: 1. active backend's `:storage/offload?` (the backend knows its own cost) 2. else `default-offload?` (engine default policy) 3. no active backend -> always inline (zero regression) Either way the bytes are COMPACTED first (`image-convert/compact`): a PNG / JPEG / GIF payload is re-compressed LOSSLESSLY -- oxipng, jpegtran-style marker stripping, gifsicle differencing -- before it becomes a BLOB or a backend PUT. That is the only transformation this rail performs, it is verified per payload (strictly smaller AND the same format at the same dimensions and frame count, else the original is kept), and it is why a matplotlib figure costs a fraction of a row.
Borrowed EYES for a blind target model.
A session routes to ONE model, and that model may have no :vision capability
(a Copilot tier without vision, glm-5-turbo, deepseek, most coding plans). The
images are still real — the user attached a screenshot, an earlier iteration
plotted a figure — so today they are dropped from the wire and the model is told
to open them with PIL. PIL answers (1920, 1080) RGB and nothing else: pixel
size is not meaning, and the agent burns a tool call to learn nothing.
This namespace closes that gap WITHOUT switching the turn's model: one cheap
side-channel ask! prefers a :vision model from the foreground provider, then
the last provider whose eyes worked, then the cheapest+fastest eyes elsewhere in
the same fleet. Svar owns that ordered provider preference and capability filter.
The pinned model, its thinking chain and its tool continuity are untouched.
Three properties make it affordable rather than wasteful:
A LEAF: svar + attachments + config + runtime-settings + toggles, never back on the loop.
Borrowed EYES for a blind target model. A session routes to ONE model, and that model may have no `:vision` capability (a Copilot tier without vision, glm-5-turbo, deepseek, most coding plans). The images are still real — the user attached a screenshot, an earlier iteration plotted a figure — so today they are dropped from the wire and the model is told to open them with PIL. PIL answers `(1920, 1080) RGB` and nothing else: pixel size is not meaning, and the agent burns a tool call to learn nothing. This namespace closes that gap WITHOUT switching the turn's model: one cheap side-channel `ask!` prefers a `:vision` model from the foreground provider, then the last provider whose eyes worked, then the cheapest+fastest eyes elsewhere in the same fleet. Svar owns that ordered provider preference and capability filter. The pinned model, its thinking chain and its tool continuity are untouched. Three properties make it affordable rather than wasteful: - CONTENT-KEYED CACHE. Attachments replay on every request for the rest of the session, so a per-request description would be re-billed forever. Keyed by the digest of the bytes, an image is described exactly ONCE per process. - OWN FAILURE POLICY. Like titling, the describe call never waits out a 429 and never fails over provider chains: the foreground turn owns the quota. A refusal or a deadline returns nil and the caller keeps today's behaviour. - SECOND-HAND BY CONTRACT. The description is labelled as another model's report in the prompt, never as the agent's own sight, so pixel-exact work still goes through the imaging path. A LEAF: svar + attachments + config + runtime-settings + toggles, never back on the loop.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |