User-message image and video attachments.
Dropping a file onto the terminal pastes its PATH into the input (the terminal's drop behavior — same mechanism pi relies on). At turn start the engine scans the user message for path-shaped tokens that resolve to real image files, reads them, and attaches them to the initial user message as multimodal content blocks. Channel-neutral: every channel gets the same behavior because the scan runs in the engine, not the channel.
Only files the model can genuinely consume are attached: the MIME type is sniffed from magic bytes (pi-parity: jpeg / non-animated png / gif / webp / bmp, plus MP4/QuickTime clips) or, for SVG, from the markup head -- never trusted from the extension alone.
Storing and SENDING are deliberately separate concerns:
max-image-bytes
is skipped with a reason the prompt assembler surfaces.wire-image is the one gate every image crosses on its way to
a provider: it decodes the payload to prove it is pixels, re-containers
what no wire accepts (BMP/SVG -> PNG via image-convert, on
com.blockether/imaging, so it behaves identically in the native image)
and REFUSES what it cannot turn into a picture.The split is what makes a bad attachment survivable. Attachments replay on every later turn, so a row blessed once on the way IN is shipped forever — one corrupt PNG that way is a permanent provider 400. Judged on the way OUT, the same row is simply dropped and the session keeps working.
User-message image and video attachments.
Dropping a file onto the terminal pastes its PATH into the input (the
terminal's drop behavior — same mechanism pi relies on). At turn start
the engine scans the user message for path-shaped tokens that resolve
to real image files, reads them, and attaches them to the initial user
message as multimodal content blocks. Channel-neutral: every channel
gets the same behavior because the scan runs in the engine,
not the channel.
Only files the model can genuinely consume are attached: the MIME type
is sniffed from magic bytes (pi-parity: jpeg / non-animated png / gif /
webp / bmp, plus MP4/QuickTime clips) or, for SVG, from the markup head --
never trusted from the extension alone.
Storing and SENDING are deliberately separate concerns:
* STORE — the original bytes under their sniffed container, nothing
converted, nothing downscaled, nothing re-compressed. What the user
supplied is what the session keeps, and a file over `max-image-bytes`
is skipped with a reason the prompt assembler surfaces.
* SEND — [[wire-image]] is the one gate every image crosses on its way to
a provider: it decodes the payload to prove it is pixels, re-containers
what no wire accepts (BMP/SVG -> PNG via `image-convert`, on
`com.blockether/imaging`, so it behaves identically in the native image)
and REFUSES what it cannot turn into a picture.
The split is what makes a bad attachment survivable. Attachments replay on
every later turn, so a row blessed once on the way IN is shipped forever —
one corrupt PNG that way is a permanent provider 400. Judged on the way OUT,
the same row is simply dropped and the session keeps working.(collect-user-images text)(collect-user-images text
{:keys [workspace-root max-bytes max-images]
:or {max-bytes max-image-bytes
max-images max-image-count}})Scan text (one user message) for paths of readable image files and
load them as attachments.
Options:
:workspace-root - base for relative candidates (default: cwd).
:max-bytes - per-image cap (default max-image-bytes).
:max-images - attachment count cap (default max-image-count).
Returns {:attached [{:path :media-type :base64 :size :size-label}] :skipped [{:path :reason}]} — :skipped names sniffed
image files that were rejected (size cap / count cap) so the prompt
assembler can tell the model WHY an image it can see referenced is
absent. Non-image or non-existent candidates are silently ignored.
Never throws — a failure to read one file skips that file.
Scan `text` (one user message) for paths of readable image files and
load them as attachments.
Options:
:workspace-root - base for relative candidates (default: cwd).
:max-bytes - per-image cap (default [[max-image-bytes]]).
:max-images - attachment count cap (default [[max-image-count]]).
Returns `{:attached [{:path :media-type :base64 :size :size-label}]
:skipped [{:path :reason}]}` — `:skipped` names sniffed
image files that were rejected (size cap / count cap) so the prompt
assembler can tell the model WHY an image it can see referenced is
absent. Non-image or non-existent candidates are silently ignored.
Never throws — a failure to read one file skips that file.(detect-image-mime b)Sniff a supported image MIME type from the leading bytes of a file.
Returns "image/png" | "image/jpeg" | "image/gif" | "image/webp" |
"image/bmp" | "image/svg+xml", or nil when the bytes are not a supported
still image. BMP and SVG are not wire-legal themselves — they are readable,
and image-convert turns them into PNG before they are sent.
Animated PNGs and JPEG-LS return nil (provider-rejected shapes).
Sniff a supported image MIME type from the leading bytes of a file. Returns "image/png" | "image/jpeg" | "image/gif" | "image/webp" | "image/bmp" | "image/svg+xml", or nil when the bytes are not a supported still image. BMP and SVG are not wire-legal themselves — they are readable, and `image-convert` turns them into PNG before they are sent. Animated PNGs and JPEG-LS return nil (provider-rejected shapes).
(detect-media-mime b)The sniffed type of anything vis can attach: detect-image-mime first,
then detect-video-mime. Stills win the tie deliberately -- HEIF/AVIF
photos share the MP4 container header.
The sniffed type of anything vis can attach: [[detect-image-mime]] first, then [[detect-video-mime]]. Stills win the tie deliberately -- HEIF/AVIF photos share the MP4 container header.
(detect-video-mime b)Sniff a supported video MIME type from the leading bytes of a file. Returns "video/mp4" | "video/quicktime", or nil.
ISO-BMFF only (ftyp at offset 4); the major brand at offset 8 decides.
qt is QuickTime, a [[still-image-brands]] brand is a PHOTO (nil, never a
clip), anything else is MP4.
Sniff a supported video MIME type from the leading bytes of a file. Returns "video/mp4" | "video/quicktime", or nil. ISO-BMFF only (`ftyp` at offset 4); the major brand at offset 8 decides. `qt ` is QuickTime, a [[still-image-brands]] brand is a PHOTO (nil, never a clip), anything else is MP4.
(display-only? attachment)True when this attachment was recorded DISPLAY-ONLY: shown in the UI and stored in the session DB, but deliberately never rendered as an image block.
The escape hatch for the one thing multimodal replay cannot undo — an image
replays IN FULL on every later request, so a screenshot the model does not
actually need is re-billed forever. vis_attach(..., display_only=True)
stamps :is-display-only, and the send-time gate routes the row to
:skipped + :readable-blind? instead: the model is TOLD the file exists
and can open the bytes on demand (vis_read_attachment) or ask for it back
with vis_reinspect_attachment.
True when this attachment was recorded DISPLAY-ONLY: shown in the UI and stored in the session DB, but deliberately never rendered as an image block. The escape hatch for the one thing multimodal replay cannot undo — an image replays IN FULL on every later request, so a screenshot the model does not actually need is re-billed forever. `vis_attach(..., display_only=True)` stamps `:is-display-only`, and the send-time gate routes the row to `:skipped` + `:readable-blind?` instead: the model is TOLD the file exists and can open the bytes on demand (`vis_read_attachment`) or ask for it back with `vis_reinspect_attachment`.
Per-image byte cap. Anthropic's API limit is 5MB/image; OpenAI allows more, but the smallest common bound keeps one attachment valid on every wire.
Per-image byte cap. Anthropic's API limit is 5MB/image; OpenAI allows more, but the smallest common bound keeps one attachment valid on every wire.
Attachment count cap per user message. Guards against a pathological message (e.g. a pasted directory listing) ballooning the request.
Attachment count cap per user message. Guards against a pathological message (e.g. a pasted directory listing) ballooning the request.
Per-clip byte cap. A clip is never sent in its own container -- it leaves as a small animated GIF ([[wire-verdict]]) -- so this bounds the ORIGINAL the session keeps and the memory one drop can cost, not any provider limit. A 32MB ceiling covers a normal screen recording or a phone clip.
Per-clip byte cap. A clip is never sent in its own container -- it leaves as a small animated GIF ([[wire-verdict]]) -- so this bounds the ORIGINAL the session keeps and the memory one drop can cost, not any provider limit. A 32MB ceiling covers a normal screen recording or a phone clip.
How far past max-image-bytes a file may sit and still be worth reading for
a shrink attempt. Bounds the memory one pathological drop can cost.
How far past `max-image-bytes` a file may sit and still be worth reading for a shrink attempt. Bounds the memory one pathological drop can cost.
(prepare-inline-attachments attachments)(prepare-inline-attachments attachments
{:keys [max-bytes max-images]
:or {max-bytes max-image-bytes
max-images max-image-count}})Validate already-encoded image attachments delivered INLINE (web/API upload)
rather than as filesystem paths. Each entry is {:base64 :filename :media-type?};
the base64 may be a bare payload or a data:...;base64, URL. Decodes each,
sniffs the MIME from magic bytes (the declared :media-type is NEVER trusted),
enforces the same caps as collect-user-images, and returns the same
{:attached [...] :skipped [...]} shape so the assemble seam treats
disk-scanned and inline images uniformly. Never throws.
Like collect-user-images it stores the ORIGINAL payload under its SNIFFED
container and converts nothing: what a provider will accept is only knowable
at SEND time, against that turn's model (see wire-image).
Validate already-encoded image attachments delivered INLINE (web/API upload)
rather than as filesystem paths. Each entry is `{:base64 :filename :media-type?}`;
the base64 may be a bare payload or a `data:...;base64,` URL. Decodes each,
sniffs the MIME from magic bytes (the declared `:media-type` is NEVER trusted),
enforces the same caps as [[collect-user-images]], and returns the same
`{:attached [...] :skipped [...]}` shape so the assemble seam treats
disk-scanned and inline images uniformly. Never throws.
Like [[collect-user-images]] it stores the ORIGINAL payload under its SNIFFED
container and converts nothing: what a provider will accept is only knowable
at SEND time, against that turn's model (see [[wire-image]]).(provider-image-media-type? media-type)True when media-type is one of provider-image-media-types.
True when `media-type` is one of [[provider-image-media-types]].
The ONLY image media types a vision wire accepts VERBATIM. Anthropic names
exactly these four in its rejection (the image data you provided does not represent a valid image … supported image formats: ['image/jpeg', 'image/png', 'image/gif', 'image/webp']); OpenAI and Gemini are supersets.
Anything else — an image/svg+xml figure from vis_attach/matplotlib, a
BMP screenshot — is a hard 400, and since attachments
REPLAY on every later turn ONE such row kills the whole session.
The ONLY image media types a vision wire accepts VERBATIM. Anthropic names exactly these four in its rejection (`the image data you provided does not represent a valid image … supported image formats: ['image/jpeg', 'image/png', 'image/gif', 'image/webp']`); OpenAI and Gemini are supersets. Anything else — an `image/svg+xml` figure from `vis_attach`/matplotlib, a BMP screenshot — is a hard 400, and since attachments REPLAY on every later turn ONE such row kills the whole session.
(scan-image-descriptors text)(scan-image-descriptors text {:keys [workspace-root]})Resolve every image the user text points at, WITHOUT loading pixel bytes.
Returns [{:path :media-type :size :size-label :filename}] for files whose
magic bytes sniff to a supported still image — ordered, de-duped. Cheap
enough to run on every paste (only a small file-head read per candidate).
Never throws.
Resolve every image the user text points at, WITHOUT loading pixel bytes.
Returns `[{:path :media-type :size :size-label :filename}]` for files whose
magic bytes sniff to a supported still image — ordered, de-duped. Cheap
enough to run on every paste (only a small file-head read per candidate).
Never throws.(text->chip-preview text)One-line, path-free rendering of ONE user message for compact previews (queue rows, tab titles, notifications).
Every image-shaped path token collapses to a short name.png chip (see
text->inline-chips), the prose around it survives, and whitespace collapses
to single spaces. Pure string work — no filesystem access, no pixel bytes — so
it is safe on a paint path and on text whose files have since been deleted.
That is the whole point: a queued message authored by dropping a screenshot
used to render as a raw /var/folders/…/clipboard-….png, which tells the user
nothing about what they queued.
Returns nil when the message is nothing but images — the caller should paint attachment chips alone rather than an empty row. The ORIGINAL text is never mutated; callers keep it for re-send/edit so the paths still re-attach.
One-line, path-free rendering of ONE user message for compact previews (queue rows, tab titles, notifications). Every image-shaped path token collapses to a short `name.png` chip (see `text->inline-chips`), the prose around it survives, and whitespace collapses to single spaces. Pure string work — no filesystem access, no pixel bytes — so it is safe on a paint path and on text whose files have since been deleted. That is the whole point: a queued message authored by dropping a screenshot used to render as a raw `/var/folders/…/clipboard-….png`, which tells the user nothing about what they queued. Returns nil when the message is nothing but images — the caller should paint attachment chips alone rather than an empty row. The ORIGINAL text is never mutated; callers keep it for re-send/edit so the paths still re-attach.
(text->inline-chips text)text with every image-shaped PATH token replaced by a short name.png
chip, and NOTHING else touched — line breaks, indentation and the prose around
each path all survive.
Pure string work: no filesystem access, no pixel bytes, so it is safe on a paint path and on text whose files have since been deleted. Use it wherever a user message is DISPLAYED (transcript bubble, notification body); the original text is never mutated, so re-send/edit still ships the path that re-attaches the picture.
`text` with every image-shaped PATH token replaced by a short `name.png` chip, and NOTHING else touched — line breaks, indentation and the prose around each path all survive. Pure string work: no filesystem access, no pixel bytes, so it is safe on a paint path and on text whose files have since been deleted. Use it wherever a user message is DISPLAYED (transcript bubble, notification body); the original text is never mutated, so re-send/edit still ships the path that re-attaches the picture.
(unsupported-media-reason media-type)Why an otherwise readable image was not attached, in the user's words.
Why an otherwise readable image was not attached, in the user's words.
(video-media-type? media-type)True when media-type is one of video-media-types.
True when `media-type` is one of [[video-media-types]].
Video containers vis stores. No wire takes any of them verbatim: a clip is transcoded to an animated GIF at SEND time (see [[wire-verdict]]), which is why storing one is safe even though sending the container never is.
Video containers vis stores. No wire takes any of them verbatim: a clip is transcoded to an animated GIF at SEND time (see [[wire-verdict]]), which is why storing one is safe even though sending the container never is.
(wire-image attachment)(wire-image {:keys [media-type base64] :as attachment}
{:keys [max-bytes] :or {max-bytes max-image-bytes}})ONE stored attachment as the wire will actually carry it — the single gate every image crosses on its way to a provider, applied at SEND time.
Send time is the only correct layer. Storage is permanent and providers are
not: the model that will read a row is unknown when the row is written, an
attachment REPLAYS on every later turn, and a payload converted (or blessed)
on the way IN can never be reconsidered — which is exactly how one corrupt
83-byte PNG, wire-legal by media type and garbage in its IDAT, bricked a
whole session with a permanent Could not process image 400. Here the same
row is re-judged every turn, so a bad one is DROPPED instead of fatal and the
session heals itself.
Every container is handled in one place:
max-bytes — OPTIMIZED first (the imaging library's
oxipng / jpegtran / gifsicle pass, then its bounded ladder) and refused
only when even that does not fitReturns the attachment with :media-type/:base64/:size/:size-label
replaced by the WIRE payload, {:path :filename :reason} when it cannot be
sent, or nil when it is not an image at all. Never throws.
ONE stored attachment as the wire will actually carry it — the single gate
every image crosses on its way to a provider, applied at SEND time.
Send time is the only correct layer. Storage is permanent and providers are
not: the model that will read a row is unknown when the row is written, an
attachment REPLAYS on every later turn, and a payload converted (or blessed)
on the way IN can never be reconsidered — which is exactly how one corrupt
83-byte PNG, wire-legal by media type and garbage in its `IDAT`, bricked a
whole session with a permanent `Could not process image` 400. Here the same
row is re-judged every turn, so a bad one is DROPPED instead of fatal and the
session heals itself.
Every container is handled in one place:
* JPEG / PNG / GIF / WebP — decoded to prove they are pixels, then sent
BYTE-IDENTICAL (a header sniff is not a picture)
* SVG / SVGZ — rasterized to PNG (no wire reads markup)
* MP4 / QuickTime — sampled into an animated GIF (no wire takes a clip)
* BMP, TIFF, anything else the decoder reads — re-containered to PNG
* HEIC/AVIF, a corrupt raster, an image past the decoder's limits —
refused, with the decoder's own words
* anything over `max-bytes` — OPTIMIZED first (the imaging library's
oxipng / jpegtran / gifsicle pass, then its bounded ladder) and refused
only when even that does not fit
* a non-image artifact (csv/pdf/wav/…) — nil: not an image, not a failure
Returns the attachment with `:media-type`/`:base64`/`:size`/`:size-label`
replaced by the WIRE payload, `{:path :filename :reason}` when it cannot be
sent, or nil when it is not an image at all. Never throws.(wire-images images)(wire-images images {:keys [vision?] :or {vision? true} :as opts})wire-image over a whole user message's attachments, keeping the
{:attached [...] :skipped [{:path :reason}]} shape the prompt manifest
speaks — so an image that could not be sent is NAMED to the model instead of
vanishing.
:vision? false (a text-only target: Copilot without vision, glm-5-turbo,
deepseek …) attaches nothing and marks every image :readable-blind?: the
files are real and on disk, so the manifest can tell the model to open them
with an imaging library instead of hunting for blocks that are not there.
A DISPLAY-ONLY row (display-only?) takes that same blind path even on a
vision model, and is checked FIRST: the caller asked for the bytes to stay
off the wire, and that beats every capability question.
[[wire-image]] over a whole user message's attachments, keeping the
`{:attached [...] :skipped [{:path :reason}]}` shape the prompt manifest
speaks — so an image that could not be sent is NAMED to the model instead of
vanishing.
`:vision?` false (a text-only target: Copilot without vision, glm-5-turbo,
deepseek …) attaches nothing and marks every image `:readable-blind?`: the
files are real and on disk, so the manifest can tell the model to open them
with an imaging library instead of hunting for blocks that are not there.
A DISPLAY-ONLY row ([[display-only?]]) takes that same blind path even on a
vision model, and is checked FIRST: the caller asked for the bytes to stay
off the wire, and that beats every capability question.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |