Liking cljdoc? Tell your friends :D

Test Filter: Intelligent Test Selection for Clojure

Overview

test filter

Test Filter is an intelligent test selection tool for Clojure projects that dramatically reduces test execution time by running only tests affected by your code changes. Instead of running your entire test suite on every change, Test Filter analyzes your source code dependencies and git history to determine the minimum set of tests that need to run.

This project source and documentation was completely written by Claude Code. It wrote more than I wanted and I’ve not had time to verify the documentation. Feel free to play with it, but I make ZERO guarantees that anything works as advertised.

Key Features

  • Dependency-aware: Builds a complete symbol dependency graph using static analysis

  • Content-hash based change detection: Uses SHA256 hashes of normalized code to detect semantic changes

  • Formatting-immune: Ignores whitespace, indentation, and docstring changes - only logic changes trigger tests

  • Git-integrated: Tracks file changes between revisions to identify affected code

  • Intelligent caching: Incrementally updates analysis with content hashes, avoiding full re-scans

  • Multiple test frameworks: Supports clojure.test, fulcro-spec, and custom test macros

  • Integration test handling: Special handling for integration tests with broad dependencies

  • CLJC support: Properly handles Clojure Common files with reader conditionals

  • Flexible output: Multiple formats for integration with various test runners

How It Works

Test Filter uses a two-cache architecture to track which tests need to run:

  1. Analysis Cache (.test-filter-cache.edn): Ephemeral snapshot of current codebase

    • Contains dependency graph and current content hashes

    • Completely overwritten on each analyze! command

    • Only used to communicate between CLI steps

  2. Success Cache (.test-filter-success.edn): Persistent baseline of verified code

    • Contains content hashes of successfully tested symbols

    • Only updated when you explicitly mark tests as verified

    • Persists across sessions and builds

Workflow

  1. Analyze: Build dependency graph and generate current content hashes

    • Parses all source files with clj-kondo

    • Creates SHA256 hashes of normalized function bodies (ignoring docstrings/formatting)

    • Writes snapshot to analysis cache

  2. Select Tests: Compare current state vs. verified baseline

    • Loads current hashes from analysis cache

    • Loads verified hashes from success cache

    • Identifies symbols where current hash ≠ verified hash

    • Walks dependency graph to find affected tests

  3. Run Tests: Execute the selected tests (user action)

  4. Mark Verified: Record that tests passed

    • Updates success cache with hashes from current analysis

    • Only done after tests successfully pass

    • Creates new baseline for future comparisons

Example: Formatting Change (No Tests Triggered)
1. Analyze: generates hash "abc123" for `foo/bar` (docstring + formatting)
2. User reformats docstring
3. Analyze: generates hash "abc123" for `foo/bar` (still same - normalized)
4. Select Tests: current hash = success hash → NO tests needed
Example: Logic Change (Tests Triggered)
1. Success cache has: foo/bar → "abc123" (verified)
2. User changes `foo/bar` logic: (* x 2) becomes (* x 3)
3. Analyze: generates new hash "def456" for `foo/bar`
4. Select Tests: current hash ≠ success hash → change detected
5. Walk Graph: find tests depending on `foo/bar`
   - `baz/qux` uses `foo/bar`
   - `app/handler` uses `baz/qux`
   - `app-test/handler-test` tests `app/handler`
6. Return Selection: {:tests [app-test/handler-test]
                      :changed-symbols #{foo/bar}
                      :changed-hashes {foo/bar "def456"}}
7. User runs tests, they pass
8. Mark Verified: updates success cache with foo/bar → "def456"

Installation

Add to your deps.edn:

{:deps {com.fulcrologic/test-filter {:mvn/version "1.0.0" }}

 ;; optional
 :aliases
 {:cli {:main-opts ["-m" "com.fulcrologic.test-filter.cli"]}}}

Usage

Command Line Interface

Analyze Command

Analyze the current codebase and update the analysis cache:

clojure -M:cli analyze

This command:

  • Runs clj-kondo analysis on your source code

  • Builds a complete symbol dependency graph

  • Generates content hashes for all symbols

  • Overwrites .test-filter-cache.edn with current state

  • NOTE: Does NOT update the success cache - that’s done with mark-verified

Select Command

Find tests affected by changes:

# Basic usage
clojure -M:cli select

# With verbose output showing statistics
clojure -M:cli select -v

# Get all tests (ignore changes)
clojure -M:cli select --all

Output Formats

# Fully-qualified test vars (default)
clojure -M:cli select -o vars

# Test namespaces only
clojure -M:cli select -o namespaces

# Kaocha command-line format
clojure -M:cli select -o kaocha

Status Command

Check cache status:

clojure -M:cli status

Shows:

  • Whether analysis cache exists

  • Whether success cache exists

  • Cache file sizes and timestamps

  • Number of verified symbols (with -v flag)

Mark Verified Command

Mark tests as successfully verified (updates success cache):

# Mark all selected tests as verified
clojure -M:cli mark-verified

# Mark specific tests as verified
clojure -M:cli mark-verified -t my.app.core-test/foo-test

This command:

  • Updates .test-filter-success.edn with verified symbol hashes

  • Should only be run after tests pass

  • Creates the baseline for future test selection

Clear Command

Invalidate the caches:

# Clear analysis cache only
clojure -M:cli clear

# Clear both analysis and success caches
clojure -M:cli clear --all

Programmatic API

Use Test Filter from the REPL or your code:

(require '[com.fulcrologic.test-filter.core :as tf])

;; 1. Analyze the codebase (generates current state)
(tf/analyze! :paths ["src/main" "src/test"])

;; 2. Select tests based on changes
(def selection (tf/select-tests :verbose true))

;; The selection contains everything needed for verification
selection
;; => {:tests [my.app-test/foo-test my.app-test/bar-test]
;;     :changed-symbols #{my.app/foo my.app/bar}
;;     :changed-hashes {my.app/foo "abc123..." my.app/bar "def456..."}
;;     :graph {...}
;;     :stats {...}}

;; 3. Show affected tests
(tf/print-tests (:tests selection) :format :namespaces)

;; 4. Run the tests (external - use your test runner)
;; ... run tests ...

;; 5. Mark verified after tests pass
(tf/mark-verified! selection)  ; Mark all selected tests
;; or
(tf/mark-verified! selection [my.app-test/foo-test])  ; Mark subset

Integration with Test Runners

Kaocha

# Run only affected tests with Kaocha
clojure -M:cli select -o kaocha | xargs clojure -M:kaocha

Cognitect test-runner

# Get affected test namespaces
TESTS=$(clojure -M:cli select -o namespaces)

# Run with test-runner
if [ -n "$TESTS" ]; then
  clojure -M:test -n $TESTS
fi

Typical Workflow

# 1. Analyze current codebase
clojure -M:cli analyze

# 2. Make code changes
# ... edit files ...

# 3. Analyze again to capture changes
clojure -M:cli analyze

# 4. Select affected tests
clojure -M:cli select -v

# 5. Run only affected tests
clojure -M:cli select -o kaocha | xargs clojure -M:kaocha

# 6. If tests pass, mark as verified
clojure -M:cli mark-verified

# 7. Continue development
# ... edit more files ...

# 8. Next iteration - analyze and select again
clojure -M:cli analyze
clojure -M:cli select -v
# ... only new changes will trigger tests ...

CI/CD Integration

#!/bin/bash
# In your CI pipeline

# Analyze current PR branch
clojure -M:cli analyze

# Select affected tests (compares against success cache from main)
TESTS=$(clojure -M:cli select -o namespaces)

if [ -n "$TESTS" ]; then
  echo "Running affected tests: $TESTS"
  clojure -M:kaocha --focus $TESTS

  # If tests pass, update success cache
  if [ $? -eq 0 ]; then
    clojure -M:cli mark-verified
    # Commit updated success cache to track verified state
    git add .test-filter-success.edn
    git commit -m "Update verified test baseline"
  fi
else
  echo "No tests affected by changes"
fi
The .test-filter-success.edn file should be committed to your repository to track the verified baseline across CI runs. The .test-filter-cache.edn file is ephemeral and should be in .gitignore.

Advanced Features

Macro-Based Test Detection

Test Filter supports test frameworks that use macros instead of deftest:

(ns my-app.spec-test
  (:require [fulcro-spec.core :refer [specification assertions]]))

(specification "User registration"
  (assertions
    "creates a new user"
    (register-user {:name "Alice"}) => {:id 1 :name "Alice"}))

Detected test frameworks:

  • fulcro-spec.core/specification

  • Custom macros (configurable)

Integration Test Handling

Integration tests often have broad dependencies. Test Filter detects them by namespace pattern (.integration.) and applies special handling:

(ns my-app.integration.api-test
  (:require [clojure.test :refer [deftest is]]
            [my-app.system :as system]))

(deftest test-user-api
  (let [sys (system/start)]
    ;; Integration test
    (is (= 200 (:status (api-call sys))))))

Options for integration tests:

  1. Conservative mode (default): Run integration tests when uncertain about dependencies

  2. Metadata targeting: Specify exact dependencies with :test-targets metadata

  3. Configuration file: External configuration for complex cases

CLJC File Support

Test Filter properly handles Clojure Common (.cljc) files with reader conditionals:

(ns my-app.utils
  #?(:clj (:import [java.nio.file Paths])))

(defn normalize-path [path]
  #?(:clj  (-> (Paths/get path (into-array String []))
               (.normalize)
               (.toString))
     :cljs (.normalize js/path path)))

  • Analyzes only the :clj side of CLJC files

  • Ignores pure .cljs files

  • Tracks dependencies correctly across platforms

Architecture

Content-Hash Based Change Detection

Test Filter uses a sophisticated approach to detect which code changes actually require testing:

  1. Parse with EDN reader: Uses clojure.tools.reader to parse code as data structures

  2. Strip docstrings: Removes documentation strings from function definitions

  3. Normalize formatting: Uses pr-str to get consistent formatting regardless of whitespace/indentation

  4. Generate SHA256 hash: Creates a unique hash representing the function’s logic

  5. Cache hashes: Stores hashes alongside the dependency graph

  6. Compare on change: When files change, re-analyze and compare new hashes to cached ones

  7. Identify real changes: Only symbols with different hashes are considered "changed"

This means:

  • ✓ Adding/changing docstrings doesn’t trigger tests

  • ✓ Reformatting code doesn’t trigger tests

  • ✓ Reordering functions doesn’t trigger tests

  • ✓ Adding comments doesn’t trigger tests

  • ✓ Only actual logic changes trigger the appropriate tests

Components

ComponentDescription

Analyzer (analyzer.clj)

Uses clj-kondo to extract var definitions, namespace definitions, and usage relationships

Graph (graph.clj)

Builds directed dependency graph using Loom library; provides traversal operations

Content (content.clj)

Extracts function bodies, normalizes them (strips docstrings/whitespace), and generates SHA256 hashes for semantic comparison

Git (git.clj)

Wraps git commands to detect which files changed between revisions

Cache (cache.clj)

Persists graph and content hashes to EDN format; handles incremental updates and cache invalidation

Core (core.clj)

Main test selection algorithm; coordinates all components

CLI (cli.clj)

Command-line interface with multiple output formats

Data Model

Symbol Node

{:symbol 'my.ns/foo
 :type :var
 :file "src/my/ns.clj"
 :line 42
 :end-line 47
 :defined-by 'defn
 :metadata {:private false
            :macro false
            :test false}}

Dependency Edge

{:from 'my.ns/foo
 :to 'other.ns/bar
 :context 'my.ns/foo}

Analysis Cache Structure

;; .test-filter-cache.edn (ephemeral, not committed to git)
{:analyzed-at "2025-01-09T10:30:00Z"
 :paths ["src/main" "src/test"]
 :nodes {symbol -> node-data}
 :edges [{:from :to :context}]
 :files {"src/my/ns.clj" {:symbols [...]}}
 :content-hashes {my.ns/foo "sha256..."
                  my.ns/bar "sha256..."}}

Success Cache Structure

;; .test-filter-success.edn (committed to git)
{my.ns/foo "sha256-of-verified-version..."
 my.ns/bar "sha256-of-verified-version..."
 my.ns-test/foo-test "sha256-of-test-when-it-passed..."}

Selection Object

;; Returned by select-tests
{:tests [my.ns-test/foo-test my.ns-test/bar-test]
 :changed-symbols #{my.ns/foo my.ns/baz}
 :changed-hashes {my.ns/foo "new-sha256..."
                  my.ns/baz "new-sha256..."}
 :trace {my.ns-test/foo-test {my.ns/foo [my.ns-test/foo-test my.ns/foo]}}
 :graph {...}  ; Full dependency graph
 :stats {:total-tests 12
         :selected-tests 2
         :changed-symbols 2
         :selection-rate "16.7%"}}

Project Status

✅ Completed Features

All planned phases (1-9) are complete:

  • Foundation and project setup

  • clj-kondo integration

  • Graph operations with Loom

  • Git integration and change detection

  • Cache persistence and incremental updates

  • Test selection algorithm

  • Command-line interface

  • Real-world testing and bug fixes

  • Macro-based test detection (fulcro-spec)

  • Integration test handling

  • CLJC file support

  • Content-hash based change detection (ignores formatting/docstrings)

Known Limitations

  1. Testing scope: Needs validation on larger codebases (>100k LOC)

  2. Dynamic requires: Conservative handling (assumes dependency)

  3. Circular dependencies: Not yet optimized

  4. ClojureScript: Not supported (by design, focuses on CLJ/CLJC)

Future Enhancements

  • Support for test.check generative tests

  • Parallel test execution planning

  • Coverage-based refinement

  • Watch mode for continuous testing

  • Configuration file for custom patterns

Contributing

Contributions are welcome! Please:

  1. Fork the repository

  2. Create a feature branch

  3. Add tests for new functionality

  4. Ensure all tests pass

  5. Submit a pull request

License

MIT License

Copyright (c) 2025

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Support

For issues, questions, or suggestions:

  • Open an issue on GitHub

  • Check existing documentation in PLAN.md and STATUS.md

  • Review code examples in namespace docstrings

Acknowledgments

Built with:

Can you improve this documentation?Edit on GitHub

cljdoc builds & hosts documentation for Clojure/Script libraries

Keyboard shortcuts
Ctrl+kJump to recent docs
←Move to previous article
→Move to next article
Ctrl+/Jump to the search field
Raise an issueBrowse cljdoc sourceChat on Slack
× close