com.blockether.svar.internal.rlm

Liking cljdoc? Tell your friends :D

Clojure only.

Recursive Language Model (RLM) for processing arbitrarily large contexts.

RLM enables an LLM to iteratively write and execute Clojure code to examine, filter, and process large contexts that exceed token limits. The LLM writes code that runs in a sandboxed SCI (Small Clojure Interpreter) environment, inspects results, and decides whether to continue iterating or return a final answer.

API

;; 1. Create environment (holds DB, config, SCI context)
(def env (rlm/create-env {:config llm-config :path "/tmp/my-rlm"}))

;; 2. Ingest documents (can call multiple times)
(rlm/ingest-to-env! env documents)
(rlm/ingest-to-env! env more-documents)

;; 3. Run queries (reuses same env)
(rlm/query-env! env "What is X?")
(rlm/query-env! env "Find Y" {:spec my-spec})

;; 4. Dispose when done
(rlm/dispose-env! env)

Key Features

Iterative code execution: LLM writes code, sees results, writes more code
FINAL termination: LLM signals completion by returning {:FINAL result}
Recursive llm-query: Code can call back to the LLM for sub-tasks
Sandboxed evaluation: Uses SCI for safe, controlled code execution
Documents: Complete structure stored exactly as-is:
Documents with metadata
Pages with page nodes (paragraphs, headings, images, tables)
TOC entries
Learnings: DB-backed meta-insights that persist across sessions
Spec support: Define output shape, validate FINAL answers
Auto-refinement: Self-critique loop improves answer quality

LLM Available Functions (in SCI sandbox)

Document search:

(list-documents) - List all stored documents
(get-document doc-id) - Get document metadata
(search-page-nodes query) - List/filter actual content
(get-page-node node-id) - Get full page node content
(list-page-nodes opts) - List page nodes with filters
(search-toc-entries query) - List/filter table of contents
(get-toc-entry entry-id) - Get TOC entry
(list-toc-entries) - List all TOC entries

Learnings:

(store-learning insight) - Store meta-insight
(search-learnings query) - Search learnings
(vote-learning id :useful/:not-useful) - Vote on learning

History:

(search-history n) - Get recent messages (default 5)
(get-history n) - Get recent messages (default 10)

Recursive Language Model (RLM) for processing arbitrarily large contexts.

RLM enables an LLM to iteratively write and execute Clojure code to examine,
filter, and process large contexts that exceed token limits. The LLM writes
code that runs in a sandboxed SCI (Small Clojure Interpreter) environment,
inspects results, and decides whether to continue iterating or return a final
answer.

## API

```clojure
;; 1. Create environment (holds DB, config, SCI context)
(def env (rlm/create-env {:config llm-config :path "/tmp/my-rlm"}))

;; 2. Ingest documents (can call multiple times)
(rlm/ingest-to-env! env documents)
(rlm/ingest-to-env! env more-documents)

;; 3. Run queries (reuses same env)
(rlm/query-env! env "What is X?")
(rlm/query-env! env "Find Y" {:spec my-spec})

;; 4. Dispose when done
(rlm/dispose-env! env)
```

## Key Features

- Iterative code execution: LLM writes code, sees results, writes more code
- FINAL termination: LLM signals completion by returning {:FINAL result}
- Recursive llm-query: Code can call back to the LLM for sub-tasks
- Sandboxed evaluation: Uses SCI for safe, controlled code execution
- Documents: Complete structure stored exactly as-is:
- Documents with metadata
- Pages with page nodes (paragraphs, headings, images, tables)
- TOC entries
- Learnings: DB-backed meta-insights that persist across sessions
- Spec support: Define output shape, validate FINAL answers
- Auto-refinement: Self-critique loop improves answer quality

## LLM Available Functions (in SCI sandbox)

Document search:
 - (list-documents) - List all stored documents
 - (get-document doc-id) - Get document metadata
 - (search-page-nodes query) - List/filter actual content
 - (get-page-node node-id) - Get full page node content
 - (list-page-nodes opts) - List page nodes with filters
 - (search-toc-entries query) - List/filter table of contents
 - (get-toc-entry entry-id) - Get TOC entry
 - (list-toc-entries) - List all TOC entries
  
  Learnings:
 - (store-learning insight) - Store meta-insight
 - (search-learnings query) - Search learnings
 - (vote-learning id :useful/:not-useful) - Vote on learning
 
 History:
 - (search-history n) - Get recent messages (default 5)
 - (get-history n) - Get recent messages (default 10)

`Ctrl`+`k`	Jump to recent docs
`←`	Move to previous article
`→`	Move to next article
`Ctrl`+`/`	Jump to the search field

com.blockether.svar.internal.rlm

API

Key Features

LLM Available Functions (in SCI sandbox)

*max-recursion-depth*clj

*rlm-ctx*clj

bytes->base64clj

CLAIM_SCHEMAclj

create-envclj

DEFAULT_RECURSION_DEPTHclj

dispose-env!clj

DOCUMENT_SCHEMAclj

ENTITY_EXTRACTION_SPECclj

ENTITY_SCHEMAclj

EVAL_TIMEOUT_MSclj

generate-qa-env!clj

ingest-to-env!clj

LEARNING_SCHEMAclj

LEGAL_ENTITY_SCHEMAclj

MAX_ITERATIONSclj

MESSAGE_HISTORY_SCHEMAclj

PAGE_NODE_SCHEMAclj

PAGE_SCHEMAclj

pprint-traceclj

print-traceclj

query-env!clj

register-env-def!clj

register-env-fn!clj

RELATIONSHIP_SCHEMAclj

SAFE_BINDINGSclj

save-qa!clj

TOC_ENTRY_SCHEMAclj

max-recursion-depth^clj

rlm-ctx^clj

bytes->base64^clj

CLAIM_SCHEMA^clj

create-env^clj

DEFAULT_RECURSION_DEPTH^clj

dispose-env!^clj

DOCUMENT_SCHEMA^clj

ENTITY_EXTRACTION_SPEC^clj

ENTITY_SCHEMA^clj

EVAL_TIMEOUT_MS^clj

generate-qa-env!^clj

ingest-to-env!^clj

LEARNING_SCHEMA^clj

LEGAL_ENTITY_SCHEMA^clj

MAX_ITERATIONS^clj

MESSAGE_HISTORY_SCHEMA^clj

PAGE_NODE_SCHEMA^clj

PAGE_SCHEMA^clj

pprint-trace^clj

print-trace^clj

query-env!^clj

register-env-def!^clj

register-env-fn!^clj

RELATIONSHIP_SCHEMA^clj

SAFE_BINDINGS^clj

save-qa!^clj

TOC_ENTRY_SCHEMA^clj