Automatic image shrinking for store-bound attachments.
Images are HOT (see attachment-storage): they never offload to an external
backend, so every screenshot a user drops and every plot the sandbox renders
lands inline in the session_attachment BLOB and is replayed to the vision
model for as long as its iteration stays live. A 5 MB retina PNG therefore
costs disk forever AND upload bandwidth every turn -- while the provider
downscales it server-side to ~1568px on the long edge anyway, so the extra
pixels buy the model nothing.
This namespace re-encodes such payloads before they are stored:
default-max-dimension
(the provider's own bound -- no visual information is lost that the
model would have seen),default-min-gain),
otherwise the original bytes pass through untouched.Work runs on a small daemon pool with LOW priority, off the caller's thread:
a turn with several attachments shrinks them concurrently and never waits
longer than *timeout-ms*. Every failure mode -- an undecodable payload, a
missing AWT/ImageIO stack (GraalVM native-image on macOS), a slow encode --
degrades to the original attachment, never to an exception.
Automatic image shrinking for store-bound attachments.
Images are HOT (see `attachment-storage`): they never offload to an external
backend, so every screenshot a user drops and every plot the sandbox renders
lands inline in the `session_attachment` BLOB and is replayed to the vision
model for as long as its iteration stays live. A 5 MB retina PNG therefore
costs disk forever AND upload bandwidth every turn -- while the provider
downscales it server-side to ~1568px on the long edge anyway, so the extra
pixels buy the model nothing.
This namespace re-encodes such payloads before they are stored:
1. downscale so the long edge is at most `default-max-dimension`
(the provider's own bound -- no visual information is lost that the
model would have seen),
2. re-encode -- JPEG when the image is fully opaque (screenshots, plots),
PNG when it genuinely carries transparency,
3. keep the result ONLY when it is meaningfully smaller (`default-min-gain`),
otherwise the original bytes pass through untouched.
Work runs on a small daemon pool with LOW priority, off the caller's thread:
a turn with several attachments shrinks them concurrently and never waits
longer than `*timeout-ms*`. Every failure mode -- an undecodable payload, a
missing AWT/ImageIO stack (GraalVM native-image on macOS), a slow encode --
degrades to the original attachment, never to an exception.Kill switch -- bind false to store payloads verbatim.
Kill switch -- bind false to store payloads verbatim.
Payloads below this never earn a decode/encode round-trip (64 KiB).
Payloads below this never earn a decode/encode round-trip (64 KiB).
JPEG quality for opaque images. High enough that UI text in a screenshot stays legible to a vision model.
JPEG quality for opaque images. High enough that UI text in a screenshot stays legible to a vision model.
Long-edge pixel bound. Anthropic resizes anything larger server-side and OpenAI's high-detail tiling caps out in the same neighbourhood, so pixels above this line are paid for and then thrown away.
Long-edge pixel bound. Anthropic resizes anything larger server-side and OpenAI's high-detail tiling caps out in the same neighbourhood, so pixels above this line are paid for and then thrown away.
The re-encoded payload is kept only when it is at most this fraction of the original. A 5% win is not worth a lossy round-trip.
The re-encoded payload is kept only when it is at most this fraction of the original. A 5% win is not worth a lossy round-trip.
Longest a caller waits for the background pool before shipping originals.
Longest a caller waits for the background pool before shipping originals.
Raster formats worth re-encoding. GIF is excluded: it may be animated and a single-frame re-encode would silently drop the animation.
Raster formats worth re-encoding. GIF is excluded: it may be animated and a single-frame re-encode would silently drop the animation.
(optimize data media-type)(optimize data
media-type
{:keys [max-dimension floor-bytes jpeg-quality min-gain]})Shrink one image payload. Returns nil when the bytes are already as good as
vis can make them (too small, undecodable, unsupported format, no ImageIO,
or the re-encode did not beat :min-gain); otherwise
{:bytes :media-type :size :original-size :width :height :original-width :original-height}
Pure with respect to the caller: data is never mutated. Never throws.
Shrink one image payload. Returns nil when the bytes are already as good as
vis can make them (too small, undecodable, unsupported format, no ImageIO,
or the re-encode did not beat `:min-gain`); otherwise
{:bytes :media-type :size :original-size :width :height
:original-width :original-height}
Pure with respect to the caller: `data` is never mutated. Never throws.(optimize-attachment att)Store-bound attachment envelope (:base64 :media-type :size, optionally
:filename) -> the same envelope with smaller bytes, or the SAME MAP when
nothing was gained. Adds no keys: the wire shape is untouched.
Store-bound attachment envelope (`:base64` `:media-type` `:size`, optionally `:filename`) -> the same envelope with smaller bytes, or the SAME MAP when nothing was gained. Adds no keys: the wire shape is untouched.
(optimize-attachments atts)Shrink every image in a store-bound attachment seq, concurrently, on the
background pool. Returns a vector in the SAME order. A payload that is still
in flight after *timeout-ms* (total, across the batch) is shipped as-is, so
this can delay a persist by at most that budget and never fails a turn.
Shrink every image in a store-bound attachment seq, concurrently, on the background pool. Returns a vector in the SAME order. A payload that is still in flight after `*timeout-ms*` (total, across the batch) is shipped as-is, so this can delay a persist by at most that budget and never fails a turn.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |