User-message image and video attachments.
Dropping a file onto the terminal pastes its PATH into the input (the terminal's drop behavior — same mechanism pi relies on). At turn start the engine scans the user message for path-shaped tokens that resolve to real image files, reads them, and attaches them to the initial user message as multimodal content blocks. Channel-neutral: every channel gets the same behavior because the scan runs in the engine, not the channel.
Only files the model can genuinely consume are attached: the MIME type is sniffed from magic bytes (pi-parity: jpeg / non-animated png / gif / webp / bmp, plus MP4/QuickTime clips) or, for SVG, from the markup head -- never trusted from the extension alone.
Storing and SENDING are deliberately separate concerns:
max-image-bytes
is skipped with a reason the prompt assembler surfaces.wire-image is the one gate every image crosses on its way to
a provider: it decodes the payload to prove it is pixels, re-containers
what no wire accepts (BMP/SVG -> PNG via image-convert, on
com.blockether/imaging, so it behaves identically in the native image)
and REFUSES what it cannot turn into a picture.The split is what makes a bad attachment survivable. Attachments replay on every later turn, so a row blessed once on the way IN is shipped forever — one corrupt PNG that way is a permanent provider 400. Judged on the way OUT, the same row is simply dropped and the session keeps working.
User-message image and video attachments.
Dropping a file onto the terminal pastes its PATH into the input (the
terminal's drop behavior — same mechanism pi relies on). At turn start
the engine scans the user message for path-shaped tokens that resolve
to real image files, reads them, and attaches them to the initial user
message as multimodal content blocks. Channel-neutral: every channel
gets the same behavior because the scan runs in the engine,
not the channel.
Only files the model can genuinely consume are attached: the MIME type
is sniffed from magic bytes (pi-parity: jpeg / non-animated png / gif /
webp / bmp, plus MP4/QuickTime clips) or, for SVG, from the markup head --
never trusted from the extension alone.
Storing and SENDING are deliberately separate concerns:
* STORE — the original bytes under their sniffed container, nothing
converted, nothing downscaled, nothing re-compressed. What the user
supplied is what the session keeps, and a file over `max-image-bytes`
is skipped with a reason the prompt assembler surfaces.
* SEND — [[wire-image]] is the one gate every image crosses on its way to
a provider: it decodes the payload to prove it is pixels, re-containers
what no wire accepts (BMP/SVG -> PNG via `image-convert`, on
`com.blockether/imaging`, so it behaves identically in the native image)
and REFUSES what it cannot turn into a picture.
The split is what makes a bad attachment survivable. Attachments replay on
every later turn, so a row blessed once on the way IN is shipped forever —
one corrupt PNG that way is a permanent provider 400. Judged on the way OUT,
the same row is simply dropped and the session keeps working.(collect-user-images text)(collect-user-images text
{:keys [workspace-root max-bytes max-images]
:or {max-bytes max-image-bytes
max-images max-image-count}})Scan text (one user message) for paths of readable image files and
load them as attachments.
Options:
:workspace-root - base for relative candidates (default: cwd).
:max-bytes - per-image cap (default max-image-bytes).
:max-images - attachment count cap (default max-image-count).
Returns {:attached [{:path :media-type :base64 :size :size-label}] :skipped [{:path :reason}]} — :skipped names sniffed
image files that were rejected (size cap / count cap) so the prompt
assembler can tell the model WHY an image it can see referenced is
absent. Non-image or non-existent candidates are silently ignored.
Never throws — a failure to read one file skips that file.
Scan `text` (one user message) for paths of readable image files and
load them as attachments.
Options:
:workspace-root - base for relative candidates (default: cwd).
:max-bytes - per-image cap (default [[max-image-bytes]]).
:max-images - attachment count cap (default [[max-image-count]]).
Returns `{:attached [{:path :media-type :base64 :size :size-label}]
:skipped [{:path :reason}]}` — `:skipped` names sniffed
image files that were rejected (size cap / count cap) so the prompt
assembler can tell the model WHY an image it can see referenced is
absent. Non-image or non-existent candidates are silently ignored.
Never throws — a failure to read one file skips that file.(detect-image-mime b)Sniff a supported image MIME type from the leading bytes of a file.
Returns "image/png" | "image/jpeg" | "image/gif" | "image/webp" |
"image/bmp" | "image/svg+xml", or nil when the bytes are not a supported
still image. BMP and SVG are not wire-legal themselves — they are readable,
and image-convert turns them into PNG before they are sent.
Animated PNGs and JPEG-LS return nil (provider-rejected shapes).
Sniff a supported image MIME type from the leading bytes of a file. Returns "image/png" | "image/jpeg" | "image/gif" | "image/webp" | "image/bmp" | "image/svg+xml", or nil when the bytes are not a supported still image. BMP and SVG are not wire-legal themselves — they are readable, and `image-convert` turns them into PNG before they are sent. Animated PNGs and JPEG-LS return nil (provider-rejected shapes).
(detect-media-mime b)The sniffed type of anything vis can attach: detect-image-mime first,
then detect-video-mime. Stills win the tie deliberately -- HEIF/AVIF
photos share the MP4 container header.
The sniffed type of anything vis can attach: [[detect-image-mime]] first, then [[detect-video-mime]]. Stills win the tie deliberately -- HEIF/AVIF photos share the MP4 container header.
(detect-video-mime b)Sniff a supported video MIME type from the leading bytes of a file. Returns "video/mp4" | "video/quicktime", or nil.
ISO-BMFF only (ftyp at offset 4); the major brand at offset 8 decides.
qt is QuickTime, a [[still-image-brands]] brand is a PHOTO (nil, never a
clip), anything else is MP4.
Sniff a supported video MIME type from the leading bytes of a file. Returns "video/mp4" | "video/quicktime", or nil. ISO-BMFF only (`ftyp` at offset 4); the major brand at offset 8 decides. `qt ` is QuickTime, a [[still-image-brands]] brand is a PHOTO (nil, never a clip), anything else is MP4.
(display-only? attachment)True when this attachment was recorded DISPLAY-ONLY: shown in the UI and stored in the session DB, but deliberately never rendered as an image block.
The escape hatch for the one thing multimodal replay cannot undo — an image
replays IN FULL on every later request, so a screenshot the model does not
actually need is re-billed forever. vis_attach(..., display_only=True)
stamps :is-display-only, and the send-time gate routes the row to
:skipped + :readable-blind? instead: the model is TOLD the file exists
and can open the bytes on demand (vis_read_attachment) or ask for it back
with vis_reinspect_attachment.
True when this attachment was recorded DISPLAY-ONLY: shown in the UI and stored in the session DB, but deliberately never rendered as an image block. The escape hatch for the one thing multimodal replay cannot undo — an image replays IN FULL on every later request, so a screenshot the model does not actually need is re-billed forever. `vis_attach(..., display_only=True)` stamps `:is-display-only`, and the send-time gate routes the row to `:skipped` + `:readable-blind?` instead: the model is TOLD the file exists and can open the bytes on demand (`vis_read_attachment`) or ask for it back with `vis_reinspect_attachment`.
Per-image cap on the WIRE PAYLOAD, in bytes of BASE64 -- which is what a
provider measures, not the picture that decodes out of it (see
wire-byte-budget, so the picture itself gets 3/4 of this).
Anthropic's API limit is 5MB/image and it is enforced on the base64 string
(image exceeds 5 MB maximum: 5994492 bytes > 5242880 bytes); OpenAI and
Gemini bound the whole request far more generously. The smallest common
bound keeps one attachment valid on every wire.
Per-image cap on the WIRE PAYLOAD, in bytes of BASE64 -- which is what a provider measures, not the picture that decodes out of it (see `wire-byte-budget`, so the picture itself gets 3/4 of this). Anthropic's API limit is 5MB/image and it is enforced on the base64 string (`image exceeds 5 MB maximum: 5994492 bytes > 5242880 bytes`); OpenAI and Gemini bound the whole request far more generously. The smallest common bound keeps one attachment valid on every wire.
Attachment count cap per user message. Guards against a pathological message (e.g. a pasted directory listing) ballooning the request.
Attachment count cap per user message. Guards against a pathological message (e.g. a pasted directory listing) ballooning the request.
Per-still INTAKE cap (25MB): the largest image the gateway accepts and the
session stores. Distinct from max-image-bytes, which is the PROVIDER cap
applied to the bytes that actually go on the wire.
Per-still INTAKE cap (25MB): the largest image the gateway accepts and the session stores. Distinct from [[max-image-bytes]], which is the PROVIDER cap applied to the bytes that actually go on the wire.
Per-clip byte cap. A clip is never sent in its own container -- it leaves as a small animated GIF ([[wire-verdict]]) -- so this bounds the ORIGINAL the session keeps and the memory one drop can cost, not any provider limit. A 32MB ceiling covers a normal screen recording or a phone clip.
Per-clip byte cap. A clip is never sent in its own container -- it leaves as a small animated GIF ([[wire-verdict]]) -- so this bounds the ORIGINAL the session keeps and the memory one drop can cost, not any provider limit. A 32MB ceiling covers a normal screen recording or a phone clip.
How far past max-image-bytes a file may sit and still be worth reading for
a shrink attempt. Bounds the memory one pathological drop can cost.
How far past `max-image-bytes` a file may sit and still be worth reading for a shrink attempt. Bounds the memory one pathological drop can cost.
(prepare-inline-attachments attachments)(prepare-inline-attachments attachments
{:keys [max-bytes max-images]
:or {max-bytes max-image-bytes
max-images max-image-count}})Validate already-encoded image attachments delivered INLINE (web/API upload)
rather than as filesystem paths. Each entry is {:base64 :filename :media-type?};
the base64 may be a bare payload or a data:...;base64, URL. Decodes each,
sniffs the MIME from magic bytes (the declared :media-type is NEVER trusted),
enforces the same caps as collect-user-images, and returns the same
{:attached [...] :skipped [...]} shape so the assemble seam treats
disk-scanned and inline images uniformly. Never throws.
Like collect-user-images it stores the ORIGINAL payload under its SNIFFED
container and converts nothing: what a provider will accept is only knowable
at SEND time, against that turn's model (see wire-image).
Validate already-encoded image attachments delivered INLINE (web/API upload)
rather than as filesystem paths. Each entry is `{:base64 :filename :media-type?}`;
the base64 may be a bare payload or a `data:...;base64,` URL. Decodes each,
sniffs the MIME from magic bytes (the declared `:media-type` is NEVER trusted),
enforces the same caps as [[collect-user-images]], and returns the same
`{:attached [...] :skipped [...]}` shape so the assemble seam treats
disk-scanned and inline images uniformly. Never throws.
Like [[collect-user-images]] it stores the ORIGINAL payload under its SNIFFED
container and converts nothing: what a provider will accept is only knowable
at SEND time, against that turn's model (see [[wire-image]]).(provider-image-media-type? media-type)True when media-type is one of provider-image-media-types.
True when `media-type` is one of [[provider-image-media-types]].
The ONLY image media types a vision wire accepts VERBATIM. Anthropic names
exactly these four in its rejection (the image data you provided does not represent a valid image … supported image formats: ['image/jpeg', 'image/png', 'image/gif', 'image/webp']); OpenAI and Gemini are supersets.
Anything else — an image/svg+xml figure from vis_attach/matplotlib, a
BMP screenshot — is a hard 400, and since attachments
REPLAY on every later turn ONE such row kills the whole session.
The ONLY image media types a vision wire accepts VERBATIM. Anthropic names exactly these four in its rejection (`the image data you provided does not represent a valid image … supported image formats: ['image/jpeg', 'image/png', 'image/gif', 'image/webp']`); OpenAI and Gemini are supersets. Anything else — an `image/svg+xml` figure from `vis_attach`/matplotlib, a BMP screenshot — is a hard 400, and since attachments REPLAY on every later turn ONE such row kills the whole session.
(scan-image-descriptors text)(scan-image-descriptors text {:keys [workspace-root]})Resolve every image the user text points at, WITHOUT loading pixel bytes.
Returns [{:path :media-type :size :size-label :filename}] for files whose
magic bytes sniff to a supported still image — ordered, de-duped. Cheap
enough to run on every paste (only a small file-head read per candidate).
Never throws.
Resolve every image the user text points at, WITHOUT loading pixel bytes.
Returns `[{:path :media-type :size :size-label :filename}]` for files whose
magic bytes sniff to a supported still image — ordered, de-duped. Cheap
enough to run on every paste (only a small file-head read per candidate).
Never throws.(text->chip-preview text)One-line, path-free rendering of ONE user message for compact previews (queue rows, tab titles, notifications).
Every image-shaped path token collapses to a short name.png chip (see
text->inline-chips), the prose around it survives, and whitespace collapses
to single spaces. Pure string work — no filesystem access, no pixel bytes — so
it is safe on a paint path and on text whose files have since been deleted.
That is the whole point: a queued message authored by dropping a screenshot
used to render as a raw /var/folders/…/clipboard-….png, which tells the user
nothing about what they queued.
Returns nil when the message is nothing but images — the caller should paint attachment chips alone rather than an empty row. The ORIGINAL text is never mutated; callers keep it for re-send/edit so the paths still re-attach.
One-line, path-free rendering of ONE user message for compact previews (queue rows, tab titles, notifications). Every image-shaped path token collapses to a short `name.png` chip (see `text->inline-chips`), the prose around it survives, and whitespace collapses to single spaces. Pure string work — no filesystem access, no pixel bytes — so it is safe on a paint path and on text whose files have since been deleted. That is the whole point: a queued message authored by dropping a screenshot used to render as a raw `/var/folders/…/clipboard-….png`, which tells the user nothing about what they queued. Returns nil when the message is nothing but images — the caller should paint attachment chips alone rather than an empty row. The ORIGINAL text is never mutated; callers keep it for re-send/edit so the paths still re-attach.
(text->inline-chips text)text with every image-shaped PATH token replaced by a short name.png
chip, and NOTHING else touched — line breaks, indentation and the prose around
each path all survive.
Pure string work: no filesystem access, no pixel bytes, so it is safe on a paint path and on text whose files have since been deleted. Use it wherever a user message is DISPLAYED (transcript bubble, notification body); the original text is never mutated, so re-send/edit still ships the path that re-attaches the picture.
`text` with every image-shaped PATH token replaced by a short `name.png` chip, and NOTHING else touched — line breaks, indentation and the prose around each path all survive. Pure string work: no filesystem access, no pixel bytes, so it is safe on a paint path and on text whose files have since been deleted. Use it wherever a user message is DISPLAYED (transcript bubble, notification body); the original text is never mutated, so re-send/edit still ships the path that re-attaches the picture.
(unsupported-media-reason media-type)Why an otherwise readable image was not attached, in the user's words.
Why an otherwise readable image was not attached, in the user's words.
How far past max-image-bytes a STILL may sit and still be accepted for
storage. A phone photo or a retina screenshot is routinely tens of megabytes
while the provider ceiling is 5MB, and wire-image shrinks whatever it is
given on the way OUT — so refusing the upload only loses the picture.
How far past `max-image-bytes` a STILL may sit and still be accepted for storage. A phone photo or a retina screenshot is routinely tens of megabytes while the provider ceiling is 5MB, and [[wire-image]] shrinks whatever it is given on the way OUT — so refusing the upload only loses the picture.
(video-media-type? media-type)True when media-type is one of video-media-types.
True when `media-type` is one of [[video-media-types]].
Video containers vis stores. No wire takes any of them verbatim: a clip is transcoded to an animated GIF at SEND time (see [[wire-verdict]]), which is why storing one is safe even though sending the container never is.
Video containers vis stores. No wire takes any of them verbatim: a clip is transcoded to an animated GIF at SEND time (see [[wire-verdict]]), which is why storing one is safe even though sending the container never is.
(wire-image attachment)(wire-image {:keys [media-type base64] :as attachment}
{:keys [max-bytes max-dimension]
:or {max-bytes max-image-bytes
max-dimension image-convert/max-wire-dimension}})ONE stored attachment as the wire will actually carry it — the single gate every image crosses on its way to a provider, applied at SEND time.
Send time is the only correct layer. Storage is permanent and providers are
not: the model that will read a row is unknown when the row is written, an
attachment REPLAYS on every later turn, and a payload converted (or blessed)
on the way IN can never be reconsidered — which is exactly how one corrupt
83-byte PNG, wire-legal by media type and garbage in its IDAT, bricked a
whole session with a permanent Could not process image 400. Here the same
row is re-judged every turn, so a bad one is DROPPED instead of fatal and the
session heals itself.
Every container is handled in one place:
max-dimension — DOWNSCALED to fit, because a provider
refuses an oversized picture outright once a request carries many images
(image-convert/max-wire-dimension), whatever it weighsmax-bytes — OPTIMIZED first (the
imaging library's oxipng / jpegtran / gifsicle pass, then its bounded
ladder) and refused only when even that does not fit. The cap is on the
encoded string every wire actually weighs, so the picture is held to 3/4
of itReturns the attachment with :media-type/:base64/:size/:size-label
replaced by the WIRE payload, {:path :filename :reason} when it cannot be
sent, or nil when it is not an image at all. Never throws.
ONE stored attachment as the wire will actually carry it — the single gate
every image crosses on its way to a provider, applied at SEND time.
Send time is the only correct layer. Storage is permanent and providers are
not: the model that will read a row is unknown when the row is written, an
attachment REPLAYS on every later turn, and a payload converted (or blessed)
on the way IN can never be reconsidered — which is exactly how one corrupt
83-byte PNG, wire-legal by media type and garbage in its `IDAT`, bricked a
whole session with a permanent `Could not process image` 400. Here the same
row is re-judged every turn, so a bad one is DROPPED instead of fatal and the
session heals itself.
Every container is handled in one place:
* JPEG / PNG / GIF / WebP — decoded to prove they are pixels, then sent
BYTE-IDENTICAL (a header sniff is not a picture)
* SVG / SVGZ — rasterized to PNG (no wire reads markup)
* MP4 / QuickTime — sampled into an animated GIF (no wire takes a clip)
* BMP, TIFF, anything else the decoder reads — re-containered to PNG
* HEIC/AVIF, a corrupt raster, an image past the decoder's limits —
refused, with the decoder's own words
* any side over `max-dimension` — DOWNSCALED to fit, because a provider
refuses an oversized picture outright once a request carries many images
(`image-convert/max-wire-dimension`), whatever it weighs
* anything whose BASE64 payload passes `max-bytes` — OPTIMIZED first (the
imaging library's oxipng / jpegtran / gifsicle pass, then its bounded
ladder) and refused only when even that does not fit. The cap is on the
encoded string every wire actually weighs, so the picture is held to 3/4
of it
* a non-image artifact (csv/pdf/wav/…) — nil: not an image, not a failure
Returns the attachment with `:media-type`/`:base64`/`:size`/`:size-label`
replaced by the WIRE payload, `{:path :filename :reason}` when it cannot be
sent, or nil when it is not an image at all. Never throws.(wire-images images)(wire-images images {:keys [vision?] :or {vision? true} :as opts})wire-image over a whole user message's attachments, keeping the
{:attached [...] :skipped [{:path :reason}]} shape the prompt manifest
speaks — so an image that could not be sent is NAMED to the model instead of
vanishing.
:vision? false (a text-only target: Copilot without vision, glm-5-turbo,
deepseek …) attaches nothing and marks every image :readable-blind?: the
files are real and on disk, so the manifest can tell the model to open them
with an imaging library instead of hunting for blocks that are not there.
A DISPLAY-ONLY row (display-only?) takes that same blind path even on a
vision model, and is checked FIRST: the caller asked for the bytes to stay
off the wire, and that beats every capability question.
[[wire-image]] over a whole user message's attachments, keeping the
`{:attached [...] :skipped [{:path :reason}]}` shape the prompt manifest
speaks — so an image that could not be sent is NAMED to the model instead of
vanishing.
`:vision?` false (a text-only target: Copilot without vision, glm-5-turbo,
deepseek …) attaches nothing and marks every image `:readable-blind?`: the
files are real and on disk, so the manifest can tell the model to open them
with an imaging library instead of hunting for blocks that are not there.
A DISPLAY-ONLY row ([[display-only?]]) takes that same blind path even on a
vision model, and is checked FIRST: the caller asked for the bytes to stay
off the wire, and that beats every capability question.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |