Clojask

Clojure data processing framework with parallel computing on larger-than-memory datasets

Features

• Unlimited Size

  It supports datasets larger than memory.

• Various Operations

  Although Clojask is designed for larger-than-memory datasets, like NoSQLs, it does not sacrifice common operations on relational dataframes, such as group by, aggregate, join.

• Fast

  Faster than Dask in most operations, and the larger the dataframe is, the bigger the advantage. Please find the benchmarks here.

• All Native Types

  All the datatypes used to store data are native Clojure (or Java) types.

• From File to File

  Integrate IO inside the dataframe. No need to write your own read-in and output functions.

• Parallel

  Most operations could be executed in multiple threads. See the principle in Onyx, a maintained fork of the original project.

• Lazy Operations

  Most operations will not be executed immediately. Dataframe will intelligently pipeline the operations altogether in computation.

• Little Constraints on programming

  Except for some aggregations where you need to write customized functions subject to simple templates, operations in Clojask support arbitrary Clojure functions as input

Installation

Available on Clojars .

Insert this line into your project.clj if using Leiningen.

[com.github.clojure-finance/clojask "2.0.5"]

Insert this line into your deps.edn if using CLI.

com.github.clojure-finance/clojask {:mvn/version "2.0.5"}

Requirements:

• MacOS or Linux
• JDK 17 or newer (tested on JDK 17, 21 and 25)
• The JVM flag --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED. Leiningen and deps.edn users put it in :jvm-opts; when running from plain java, pass it on the command line. Without it the first compute fails while starting the embedded media driver with IllegalAccessError: class org.agrona.UnsafeApi ... cannot access class jdk.internal.misc.Unsafe.
• The JVM flag --enable-native-access=ALL-UNNAMED, passed the same way. The lz4 compression library loads native code; JDK 24 and newer print a warning without the flag, and a future JDK will block the load.
• Each compute starts an embedded ZooKeeper on port 2188 and Aeron on port 40200, with Aeron's files in the default media-driver directory (/dev/shm on Linux). Two clojask processes on one machine need different settings: the JVM system properties clojask.zookeeper.port, clojask.aeron.port and clojask.aeron.dir, or the environment variables CLOJASK_ZOOKEEPER_PORT, CLOJASK_AERON_PORT and CLOJASK_AERON_DIR.
• Onyx writes warnings and errors to .clojask/clojask.log in the working directory, rotated at 10 MB with one backup.

Example Usage

1. Import Clojask

   (require '[clojask.dataframe :as ck])
   

2. Initialize a dataframe

   (def df (ck/dataframe "Employees-example.csv"))
   

   The source file can be found here.

   See dataframe

3. Preview the first few lines of the dataframe

   (ck/print-df df)
   

   

   See print-df

4. Change the data type of some columns

   (ck/set-type df "Salary" "double")
   (ck/set-type df "UpdateDate" "date:yyyy/MM/dd")
   (ck/print-df df)
   

   

   See set-type

5. Add 100 to Bob as NewSalary

   (ck/operate df (fn [EmployeeName Salary] (if (= EmployeeName "Bob") (+ Salary 100) Salary)) ["EmployeeName" "Salary"] "NewSalary")
   (ck/print-df df)
   

   

   See operate

6. Output the resultant dataset to "result.csv" (Use 8 threads)

   (ck/compute df 8 "result.csv" :select ["Employee" "EmployeeName" "Department" "NewSalary" "UpdateDate"])
   

   See compute

Supported Functions and Procedures

• The solid arrows point to the fixed next step; dotted arrows point to all possible next steps.
• Any step except for Initialization is optional.

Documentation

The detailed documentation for every API can be found here.

Examples

A separate repository for some typical usage of Clojask can be found here.

Problem Feedback

If your question is not answered in existing issues, feel free to create a new one.