collet.actions.lucene — io.velio/collet-action-lucene 0.2.8

compile-lucene-query clj

(compile-lucene-query data)

Given a data structure, it will compile the data structure into a lucene query string.

create-index clj

(create-index opts-or-path)

Creates or opens a Lucene IndexWriter with configurable options.

Accepts either a string path (backward compatible) or an options map.

Options: :index-path - Path to index directory (required) :ram-buffer-mb - RAM buffer size in MB (default: 256) Larger = better performance, more memory :open-mode - :create, :create-or-append, :append (default: :create-or-append) :analyzer - :standard, :whitespace, :simple, or Analyzer instance (default: :standard) :commit-on-close? - Commit on close (default: true) :use-compound-file? - Use compound format (default: false for better performance)

Returns IndexWriter (must be closed after use, prefer with-open)

Examples: ;; Simple usage (create-index "/path/to/index")

;; Performance-tuned for large file + KNN indexing (create-index {:index-path "/path/to/index" :ram-buffer-mb 512 :open-mode :create})

;; Incremental updates with code-optimized analyzer (create-index {:index-path "/path/to/index" :open-mode :append :analyzer :whitespace})

dispatch-fn clj

(dispatch-fn x)

error-map-spec clj

expression-parser clj

index! clj

(index! {:keys [index-path input docs-path] :as opts})

Indexes data into a Lucene index. Supports multiple input types:

• File path (string) - Index files from path/directory
• Dataset - Index a tech.v3.dataset directly
• Collection of maps - Index [{:a 1} {:a 2}]
• Sequence of datasets - Index multiple datasets

Parameters: :index-path - Path to Lucene index directory (required) :input - Data to index (required, see types above) :docs-path - DEPRECATED: Alias for :input when string :source-id - Custom identifier for in-memory data (default: memory://uuid)

Returns: {:indexed N :failed N :skipped N :duration-ms N :errors [...]}

Examples: ;; Index a dataset (index! {:index-path "/tmp/idx" :input (ds/->dataset [{:name "Alice"}])})

;; Index collection of maps (index! {:index-path "/tmp/idx" :input [{:name "Alice"} {:name "Bob"}]})

;; Index files (same as index-files!) (index! {:index-path "/tmp/idx" :input "/path/to/data"})

index-files! clj

(index-files! {:keys [index-path docs-path] :as opts})

Indexes structured data files (CSV, JSON-LD, Parquet, XLSX) into Lucene index.

Each row in the dataset becomes a separate Lucene document with:

• All columns as searchable/stored fields
• Full row contents as searchable text
• File metadata (path, modified, size, extension, row number)
• Optional KNN vector for semantic search

Required Options: :index-path - Path to Lucene index directory :docs-path - File or directory to index

Optional: :extensions - Set of formats to include: #{:csv :jsonld :parquet :xlsx} Default: all supported formats :exclude-patterns - Set of regex patterns to exclude files :dataset-opts - Map of options passed to tech.ml.dataset/->dataset Default behavior: Column names are automatically normalized to snake_case keywords (e.g., "First Name" -> :first_name)

                   Useful options:
                   :key-fn           - Function to transform column names
                                      (default: normalize-column-name for snake_case)
                                      Set to 'identity' to disable normalization
                                      Example: :key-fn keyword (keywords with original names)
                   :column-allowlist - Vector of column names/indices to include
                   :column-blocklist - Vector of column names/indices to exclude
                   :separator        - CSV separator character (default: comma)
                   :header-row?      - Whether CSV has headers (default: true)
                   :num-rows         - Maximum rows to load per file
                   :parser-fn        - Custom parser map {:column-name parser-fn}
                   :parser-scan-len  - Rows to scan for type detection
                   See tech.ml.dataset/->dataset for full list

:commit-every-n - Commit every N rows (default: 1000) :fail-fast? - Stop on first error (default: false, continue) :progress-fn - Callback fn(stats) for progress reporting :knn-vector-fn - fn(row-map) -> float-array for vector search

Index Writer Options (passed to create-index): :ram-buffer-mb - RAM buffer size MB (default: 256) :open-mode - :create, :create-or-append, :append :analyzer - :standard, :whitespace, :simple

Returns: {:indexed 1234 ;; Total rows indexed :failed 2 ;; Failed files :skipped 1 ;; Skipped files :duration-ms 5432 ;; Total time :errors [...]} ;; Error details

Examples:

;; Index all CSV files in directory (index-files! {:index-path "/tmp/data-index" :docs-path "/path/to/csv-files"})

;; Index with progress reporting (index-files! {:index-path "/tmp/data-index" :docs-path "/path/to/data" :extensions #{:csv :parquet} :progress-fn (fn [{:keys [indexed failed]}] (println "Progress:" indexed "rows indexed," failed "files failed"))})

;; With KNN vectors for semantic search (index-files! {:index-path "/tmp/semantic-index" :docs-path "/path/to/data" :knn-vector-fn (fn [row] ;; Combine relevant columns and get embedding (let [text (str (:title row) " " (:description row))] (get-embedding text)))})

;; Performance tuned (index-files! {:index-path "/tmp/large-index" :docs-path "/path/to/large-dataset" :ram-buffer-mb 512 :commit-every-n 5000})

;; Gzipped JSON files (automatically supported) (index-files! {:index-path "/tmp/compressed-index" :docs-path "/path/to/data" :extensions #{:jsonld}}) ;; Will index .json, .jsonld, .json.gz, .jsonld.gz

;; Mixed compressed CSV and JSON (index-files! {:index-path "/tmp/mixed-index" :docs-path "/path/to/data" :extensions #{:csv :jsonld}}) ;; Handles .csv, .csv.gz, .json, .json.gz, .jsonld, .jsonld.gz

;; Exclude compressed files if needed (index-files! {:index-path "/tmp/uncompressed-only" :docs-path "/path/to/data" :exclude-patterns #{#".*.gz$"}}) ;; Skip all .gz files

index-files-params-spec clj

index-files-return-spec clj

index-input-spec clj

Specification for indexable input data

index-writer-params-spec clj

regex? clj

registry clj

search clj

(search {:keys [index-path query default-field analyzer searcher]
         :or {default-field "contents" analyzer :standard}
         :as opts})

Search a Lucene index with QueryParser strings or Query objects.

Required options: :index-path - Path to Lucene index directory :query - Query (string for QueryParser, or Query object)

Common options: :limit - Max results to return (default: 10) :offset - Skip first N results (default: 0) :fields - Vector of field names to return (default: all stored) :default-field - Field for string queries (default: 'contents') :include-score? - Include relevance score (default: true)

Advanced options: :analyzer - :standard, :whitespace, :simple, or Analyzer instance (default: :standard) :searcher - Reuse IndexSearcher instance (advanced, optional)

Returns: tech.v3.dataset where each row contains:

• All document fields (e.g., :customer_id, :first_name, etc.)
• :score - Relevance score (if include-score? true, default)
• :rank - Position in results (1, 2, 3, ...)
• :doc_id - Internal Lucene document ID

Example: ;; Simple search (search {:index-path "/tmp/my-index" :query "laptop" :limit 10}) ;; Returns dataset: ;; | :title | :price | :score | :rank | :doc_id | ;; |----------|--------|--------|-------|---------| ;; | Laptop A | 999 | 2.5 | 1 | 42 | ;; | Laptop B | 1200 | 2.1 | 2 | 57 |

;; Field-specific search (search {:index-path "/tmp/my-index" :query "title:laptop AND price:[500 TO 1500]" :limit 20})

;; With Query object (import '[org.apache.lucene.search TermQuery] '[org.apache.lucene.index Term]) (search {:index-path "/tmp/my-index" :query (TermQuery. (Term. "category" "electronics"))})

search-one clj

(search-one opts)

Convenience function to search for a single document.

Same parameters as search, but returns a single result map or nil instead of a result collection.

Automatically sets :limit to 1.

Returns: Single result map {:score ... :fields {...}} or nil if no matches

Example: (search-one {:index-path "/tmp/my-index" :query "id:12345"})

search-params-spec clj

search-result-spec clj

stringify clj multimethod

Multi-method to stringify the parsed OData parameters