tablecloth.time

Description

This library offers tools for manipulating and processing time-series data. It complements and extends the API provided by tablecloth for working with the highly performant columnar datasets of tech.ml.dataset.

Status: Under active development. The API may change.

Design Philosophy

This library is experimental and has evolved since its inception. The current approach favors composability over high-level abstraction.

In practice, this means we do not store metadata on the dataset indicating which column contains temporal data. Rather than providing functions like adjust-frequency that implicitly operate on a designated time index, we provide primitives like add-time-columns that make it easy to derive temporal fields explicitly. Users then compose these with standard tablecloth operations (group-by, aggregate, etc.).

A Note on Indexing

Previous versions of tablecloth.time implemented index-aware functions built on an indexing mechanism in tech.ml.dataset. That indexing mechanism was removed as part of TMD v7's simplification effort (discussion on Zulip).

Key points:

1. This doesn't preclude future indexing. A future version of tablecloth.time or tech.ml.dataset may reintroduce an optional mechanism for designating a temporal index.

2. The SciCloj community is experimenting with index-free approaches. The current thinking is that explicit column arguments + binary search performs well enough for most time-series workloads, without the complexity that indexing adds. As Chris Nuernberger (dtype-next author) noted: "Just sorting the dataset and using binary search will outperform most/all tree structures in this scenario."

3. Pandas-style indexing is complex. Harold observed that "there is about as much code in Pandas doing indexing stuff as there is all of the code in TMD." The removal was a deliberate choice to keep the core library focused.

For now, we prioritize explicit column arguments and composable primitives. See doc/zulip-indexing-discussion-summary.md for the full context.

Usage

(require '[tablecloth.api :as tc]
         '[tablecloth.time.api :as tct]
         '[tablecloth.time.column.api :as tct-col])

;; Add temporal fields to a dataset
(-> my-dataset
    (tct/add-time-columns :timestamp {:year "Year" 
                                      :month "Month"
                                      :day-of-week "DayOfWeek"}))

;; Slice a time range
(-> my-dataset
    (tc/order-by :timestamp)
    (tct/slice :timestamp #time/date "2024-01-01" #time/date "2024-03-31"))

;; Add lag columns for time series analysis
(-> my-dataset
    (tct/add-lags :price [1 2 3 4]))

;; Column-level operations
(tct-col/year (my-dataset :timestamp))        ; extract year
(tct-col/floor-to-month (my-dataset :timestamp))  ; truncate to month

Resampling Example

Resample half-hourly electricity data to daily averages — no magic, just composable primitives:

;; Load half-hourly Victorian electricity data
(def vic-elec (tc/dataset "data/fpp3/vic_elec.csv" {:key-fn keyword}))

;; Resample to daily averages:
;; 1. Extract date from datetime using add-time-columns
;; 2. Group and aggregate with standard tablecloth
(-> vic-elec
    (tct/add-time-columns :Time {:date-string "Day"})
    (tc/group-by ["Day"])
    (tc/mean :Demand))

The philosophy: add-time-columns extracts temporal components you need, then standard tablecloth does the rest. Explicit columns throughout — no implicit index, no magic.

See notebooks/chapter_02_time_series_graphics.clj for more examples based on Forecasting: Principles and Practice.

Development

We use a "hybrid" Leinigen/tools.deps setup. You can use either lein or clj/clojure. We chose this hybrid setup so that we get the benefits of Leinigen's suite of build-related tools, while also benefitting from the simplicity of tools.deps' approach to dependencies. For more context on the differences and trade-offs between the the two, see this post.

Linting

We use both cljfmt and clj-kondo to lint our code. To run the linters, do:

lein lint

Tests

We run tests using the midje test runner, which will run both any midje tests and any standard clojure tests:

lein midje

Releasing to Clojars

Publishing is currently manual.

1. Update the version in project.clj (release versions should not include -SNAPSHOT).
2. Run local checks:

lein lint
lein midje

3. Deploy:

lein deploy clojars

4. After deploy, bump project.clj to the next -SNAPSHOT version for ongoing development.

Contributing

Development for this project happens in the SciCloj fundamentals study group, a group focused on improving the Cojure datascience ecosystem. We tend to hang out in the #sci-fu stream on the Clojurians Zulip, and we meet regularly to coordinate, learn, and solve problems.

We eagerly invite your participation in this project. The project is currently in an experimental phase and we are approaching its development interactively as a group in a way that is driven by openness and learning. If you are interested, please reach out.

Please peruse this project's issues to get a sense of work that is ongoing for this project.

License

MIT for now, but this is basically a placeholder. Open to suggestions.