Chroma SQLite → Milvus migration.
Railway-oriented pipeline: resolve-config → read-source → resolve-cursor → connect-target → migrate-batches!
Each step returns Result (ok/err). Cursor persists after each batch for safe interrupt/resume.
Chroma SQLite → Milvus migration. Railway-oriented pipeline: resolve-config → read-source → resolve-cursor → connect-target → migrate-batches! Each step returns Result (ok/err). Cursor persists after each batch for safe interrupt/resume.
(->migrate-config kw__7775__auto__)Coerce a keyword to a MigrateConfig variant (no data fields). Returns nil if keyword is not a valid variant.
Coerce a keyword to a MigrateConfig variant (no data fields). Returns nil if keyword is not a valid variant.
(connect-target {:keys [milvus-host milvus-port milvus-collection sqlite-path
chroma-collection]
:as ctx})Step 4: Connect to Milvus and ensure collection exists with correct dimension.
Step 4: Connect to Milvus and ensure collection exists with correct dimension.
(migrate-config variant-kw__7772__auto__)(migrate-config variant-kw__7772__auto__ data__7773__auto__)Construct a MigrateConfig variant. (migrate-config variant-keyword) for enum variants (migrate-config variant-keyword data-map) for data variants
Throws ex-info for unknown variants.
Construct a MigrateConfig variant. (migrate-config variant-keyword) for enum variants (migrate-config variant-keyword data-map) for data variants Throws ex-info for unknown variants.
(migrate-config? x__7774__auto__)True if x is a MigrateConfig ADT value.
True if x is a MigrateConfig ADT value.
Config ADT for MigrateConfig. Auto-generated by defconfig.
Config ADT for MigrateConfig. Auto-generated by defconfig.
Field registry for MigrateConfig config.
Field registry for MigrateConfig config.
Malli schema for resolved MigrateConfig.
Malli schema for resolved MigrateConfig.
(read-source {:keys [sqlite-path chroma-collection skip-empty limit] :as ctx})Step 2: Read and filter entries from Chroma SQLite.
Step 2: Read and filter entries from Chroma SQLite.
(resolve-config overrides)Step 1: Resolve and validate migration config.
Step 1: Resolve and validate migration config.
(resolve-MigrateConfig)(resolve-MigrateConfig overrides)(resolve-MigrateConfig overrides opts)Resolve MigrateConfig from environment + overrides. Returns Result: (ok resolved-map) or (err :config/resolution-failed {...})
Resolve MigrateConfig from environment + overrides.
Returns Result: (ok resolved-map) or (err :config/resolution-failed {...})(resolve-resume {:keys [cursor-path batch-size total reset-cursor? entries]
:as ctx})Step 3: Determine resume point from cursor. Pure logic + cursor IO.
Step 3: Determine resume point from cursor. Pure logic + cursor IO.
(run! overrides)Run the Chroma → Milvus migration pipeline for a single collection.
Resumes automatically from cursor. Upsert makes partial-batch replay safe.
Overrides: :dry-run? — report only, no writes :limit — cap entries to migrate :reset-cursor? — discard cursor, start fresh
Run the Chroma → Milvus migration pipeline for a single collection. Resumes automatically from cursor. Upsert makes partial-batch replay safe. Overrides: :dry-run? — report only, no writes :limit — cap entries to migrate :reset-cursor? — discard cursor, start fresh
(run-all! overrides)Migrate ALL non-test Chroma collections to Milvus.
For each collection:
Overrides: same as run! plus :include-empty? — include collections with 0 entries (default: false) :collections — explicit vec of collection names (overrides auto-discovery)
Migrate ALL non-test Chroma collections to Milvus. For each collection: 1. Creates a Milvus collection with matching dimension 2. Uses per-collection cursor for safe resume 3. Skips test collections (patterns: *-test, test-*, *-e2e-*, *-test-*) Overrides: same as run! plus :include-empty? — include collections with 0 entries (default: false) :collections — explicit vec of collection names (overrides auto-discovery)
(sync! overrides)Incremental sync: migrate only entries missing from Milvus.
Reads all entries from Chroma SQLite, queries Milvus for existing IDs, and only embeds + upserts the delta. Much faster than full re-run.
Overrides: same as run! plus :sqlite-path — path to fresh Chroma SQLite backup :collections — explicit vec of collection names
Incremental sync: migrate only entries missing from Milvus. Reads all entries from Chroma SQLite, queries Milvus for existing IDs, and only embeds + upserts the delta. Much faster than full re-run. Overrides: same as run! plus :sqlite-path — path to fresh Chroma SQLite backup :collections — explicit vec of collection names
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |