Built-in sandbox SHIM: an anydoc module for the model's Python sandbox that
turns ANY document a human sent — Word (.doc .docx .odt .rtf), PDF,
EPUB, presentations (.ppt .pptx .odp) and spreadsheets (.xls .xlsx
.xlsm .xlsb .ods .csv) — into GitHub-Flavored Markdown an LLM can
actually read, headings, lists, tables and links intact.
The parser is com.blockether/imaging's cdylib (Rust anydoc), the same FFM
stack behind the PIL shim: NOTHING here shells out to a headless office
suite, unpacks a wheel, or touches java.desktop. Conversion is pure host
work; only base64 blobs and small string-keyed maps cross the strings-only
sandbox boundary.
Reading a document is one call — anydoc.to_markdown(path) — and the richer
anydoc.to_document(...) additionally returns the format that was detected,
the evidence that identified it, every embedded binary (Asset) as real
Python bytes, and — the part search is built on — the document's own
STRUCTURE: text (the same content with none of Markdown's punctuation) plus
one block per heading, paragraph, list item, table row, code block and note,
each carrying its character span into text, its line, its heading
breadcrumb, its table cells and, for a PDF, its PAGE. That is what lets a
citation say report.pdf p.7 › Revenue › row 4 instead of report.md:812.
Matching is NOT done here. anydoc.search(query, sources) is plain Python in
resources/vis-shims/anydoc.py: it folds the plain text (ligatures, accents,
soft hyphens, line-wrapped phrases), parses a real query language, ranks with
BM25 and cuts snippets — all over data the host already handed it, with no
host call of its own. The walk that FINDS a corpus stays in Python too, on
purpose: internal/sandbox-fs confines the sandbox's filesystem to the
configured roots, so enumerating a directory host-side (through the pooled
fff index, say) would hand the sandbox names it is not allowed to see.
What this namespace does own is the CACHE. Converting a PDF costs orders of
magnitude more than searching it, and a corpus gets asked more than one
question, so every conversion is memoized on the CONTENT hash of the bytes
plus the options — an LRU with a per-entry budget, the same shape as
internal/fff-index's pool. Two questions about the same 200-file corpus
convert it once.
Built-in sandbox SHIM: an `anydoc` module for the model's Python sandbox that turns ANY document a human sent — Word (`.doc` `.docx` `.odt` `.rtf`), PDF, EPUB, presentations (`.ppt` `.pptx` `.odp`) and spreadsheets (`.xls` `.xlsx` `.xlsm` `.xlsb` `.ods` `.csv`) — into GitHub-Flavored Markdown an LLM can actually read, headings, lists, tables and links intact. The parser is `com.blockether/imaging`'s cdylib (Rust `anydoc`), the same FFM stack behind the `PIL` shim: NOTHING here shells out to a headless office suite, unpacks a wheel, or touches `java.desktop`. Conversion is pure host work; only base64 blobs and small string-keyed maps cross the strings-only sandbox boundary. Reading a document is one call — `anydoc.to_markdown(path)` — and the richer `anydoc.to_document(...)` additionally returns the format that was detected, the evidence that identified it, every embedded binary (`Asset`) as real Python `bytes`, and — the part search is built on — the document's own STRUCTURE: `text` (the same content with none of Markdown's punctuation) plus one `block` per heading, paragraph, list item, table row, code block and note, each carrying its character span into `text`, its line, its heading breadcrumb, its table cells and, for a PDF, its PAGE. That is what lets a citation say `report.pdf p.7 › Revenue › row 4` instead of `report.md:812`. Matching is NOT done here. `anydoc.search(query, sources)` is plain Python in `resources/vis-shims/anydoc.py`: it folds the plain text (ligatures, accents, soft hyphens, line-wrapped phrases), parses a real query language, ranks with BM25 and cuts snippets — all over data the host already handed it, with no host call of its own. The walk that FINDS a corpus stays in Python too, on purpose: `internal/sandbox-fs` confines the sandbox's filesystem to the configured roots, so enumerating a directory host-side (through the pooled fff index, say) would hand the sandbox names it is not allowed to see. What this namespace does own is the CACHE. Converting a PDF costs orders of magnitude more than searching it, and a corpus gets asked more than one question, so every conversion is memoized on the CONTENT hash of the bytes plus the options — an LRU with a per-entry budget, the same shape as `internal/fff-index`'s pool. Two questions about the same 200-file corpus convert it once.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |