Clojask

A Clojure dataframe library for larger-than-memory datasets. Process millions of rows with lazy evaluation and parallel execution—all with familiar dataframe operations.

Why Clojask?

Feature	Description
Larger than memory	Stream data from disk—no need to fit everything in RAM
Lazy & parallel	Operations are pipelined and executed across multiple threads
Relational operations	Filter, transform, group-by, aggregate, join, with arbitrary Clojure functions
File-to-file	Read CSV in, write CSV out, with built-in IO
Native types	Works with standard Clojure and Java types
Faster than Dask	Benchmarks show significant speedups on large datasets

Quick Start

Add to project.clj:

[com.github.clojure-finance/clojask "2.1.0"]

Or deps.edn:

com.github.clojure-finance/clojask {:mvn/version "2.1.0"}

Basic example (using Employees-example.csv):

(require '[clojask.dataframe :as ck])

;; Load a CSV
(def df (ck/dataframe "Employees-example.csv"))

;; Preview the data (the second row shows each column's type)
(ck/print-df df)
;; |         Employee |     EmployeeName |       Department |           Salary |       UpdateDate |
;; |------------------+------------------+------------------+------------------+------------------|
;; | java.lang.String | java.lang.String | java.lang.String | java.lang.String | java.lang.String |
;; |                1 |            Alice |               11 |              300 |       2020/12/12 |
;; |                2 |              Bob |               11 |              600 |       2020/12/01 |
;; ...

;; Set column types
(ck/set-type df "Salary" "double")
(ck/set-type df "UpdateDate" "date:yyyy/MM/dd")

;; Transform data: give Bob a raise, as a new column
;; (an operation over several columns must write to a new column)
(ck/operate df
  (fn [name salary]
    (if (= name "Bob") (+ salary 100) salary))
  ["EmployeeName" "Salary"]
  "NewSalary")

;; Compute with 8 threads, output to file
(ck/compute df 8 "results.csv"
  :select ["Employee" "EmployeeName" "Department" "NewSalary" "UpdateDate"])
;; results.csv:
;; Employee,EmployeeName,Department,NewSalary,UpdateDate
;; 1,Alice,11,300.0,2020/12/12
;; 2,Bob,11,700.0,2020/12/01
;; ...

Operations

Solid arrows show required sequence; dotted arrows show optional paths.

Available operations:

• Create: dataframe
• Types: set-type, set-parser, set-formatter
• Columns: operate, rename-col, select-col, delete-col, reorder-col, get-col-names
• Filter: filter
• Group: group-by, aggregate
• Combine: inner-join, left-join, right-join, outer-join, rolling-join-forward, rolling-join-backward
• Output: compute (with options such as :select, :exclude, :melt, :header), print-df, preview

sort stands outside this pipeline: it sorts the dataframe's source file on disk and writes the result directly to an output file.

Requirements

• OS: macOS or Linux
• JDK: 17 or newer (tested on 17, 21, and 25)

Upgrading from 2.0.x: clojask no longer uses Onyx, so the JVM flags it needed (--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED, --enable-native-access=ALL-UNNAMED) and the clojask.zookeeper.* / clojask.aeron.* / CLOJASK_* port and directory settings can be removed.

Documentation

• API Reference — Full documentation for all functions
• Examples Repository — Real-world usage patterns
• Aggregation Functions — Built-in and custom aggregations
• Type System — Supported data types and parsing

How It Works

Operations are collected lazily and executed together when you call compute, streaming data through your transformation pipeline without loading the full dataset into memory. Each compute runs in your JVM: one thread reads the input in batches, a pool of worker threads (the number you pass to compute) applies the operations and filters, and the calling thread writes the results in input order. Group-by and joins first sort the rows into groups by key, kept in temporary files under .clojask/ (or in memory with :in-memory true); group-by and outer joins write their results in group order.

Issues & Feedback

Found a bug or have a question? Check the existing issues or open a new one.

License

MIT