Lazytest: A new test framework for Clojure

An alternative to clojure.test, aiming to be feature-rich and easily extensible.

Getting Started

Add it to your deps.edn or project.clj:

{:aliases
 {:test {:extra-deps {io.github.noahtheduke/lazytest {:mvn/version "0.2.0"}}
         :extra-paths ["test"]
         :main-opts ["-m" "lazytest.main"]}}}

In a test file:

(ns example.readme-test
  (:require [lazytest.core :refer [defdescribe describe expect it]]))

(defdescribe seq-fns-test
  (describe keep
    (it "should reject nils"
      (expect (= '(1 false 2 3) (keep any? [nil 1 false 2 3]))))
    (it "should return a sequence"
      (expect (seq (keep any? []))))))

From the command line:

$ clojure -M:test

  lazytest.readme-test
    seq-fns-test
      #'clojure.core/keep
        √ should reject nils
        × should return a sequence FAIL

lazytest.readme-test
  seq-fns-test
    #'clojure.core/keep
      should return a sequence:

Expectation failed
Expected: (seq (keep any? []))
Actual: nil
Evaluated arguments:
 * ()

in lazytest/readme_test.clj:11

Ran 2 test cases in 0.00272 seconds.
1 failure.

Usage

With the above :test alias, you run with clojure -M:test [options] where [options] are any of the below.

• -d, --dir DIR: Directory containing tests. (Defaults to test.)
• -n, --namespace NS-SYM: Test namespace to only run.
• -v, --var VAR-SYM: Run only the specified fully-qualified symbol.
• -i, --include KEYWORD: Run only test sequences or vars with this metadata keyword.
• -e, --exclude KEYWORD: Exclude test sequences or vars with this metadata keyword.
• --output OUTPUT: Output format. Can be given multiple times. (Defaults to nested.)
• --help: Print help information.
• --version: Print version information.

Why a new test framework?

clojure.test has existed since 1.1 and while it's both ubiquitous and useful, it has a number of problems:

• is tightly couples running test code and reporting on it.
• are is strictly worse than doseq or mapv.
• clojure.test/report is ^:dynamic, but that leads to being unable to combine multiple reporters at once, or libraries such as Leiningen monkey-patching it.
• Tests can't be grouped or bundled in any meaningful way (no, defining test-ns-hook does not count).
• testing calls aren't real contexts, they're just strings.
• Fixtures serve a real purpose but because they're set on the namespace, their definition is side-effecting and using them is complicated and hard to reuse.

There exists very good libraries like Expectations v2, kaocha, eftest, Nubank's matcher-combinators, Cognitect's test-runner that improve on clojure.test, but they're all still built on a fairly shaky foundation. I think it's worthwhile to explore other ways of being, other ways of doing stuff. Is a library like lazytest good? is a testing framework like this good when used in clojure? I don't know, but I'm willing to try and find out.

Other alternatives such as Midje, classic Expectations, and speclj attempted to correct some of those issues and they made good progress. However, some (such as Midje) relied on non-list style (test => expected) and most don't worked well with modern repl-driven development practices (as seen by the popularity of the aforementioned clojure.test-compatible Expectations v2).

I like the ideas put forth in Alessandra's post above about Lazytest and hope to experiment with achieving them 14 years later, while borrowing heavily from the work in both the Clojure community and test runners frameworks in other languages.

Writing tests with 'lazytest'

The primary api is found in lazytest.core namespace. It mimics the behavior-driven testing style popularized by libraries such as RSpec and Mocha.

Define tests with defdescribe, group test suites and test cases together into a suite with describe, and define test cases with it. describe can be nested. defdescribe's docstring is optional, describe and it's docstrings are not.

(defdescribe +-test "with integers"
  (it "computes the sum of 1 and 2"
    (expect (= 3 (+ 1 2))))
  (it "computes the sum of 3 and 4"
    (assert (= 7 (+ 3 4)))))

The expect macro is like assert but carries more information about the failure, such as the given form, the returned value, and the location of the call. It throws an exception if the expression does not evaluate to logical true.

If an it runs to completion without throwing an exception, the test case is considered to have passed.

Focusing on Individual Tests and Suites

All of the test suite and test case macros (defdescribe, describe, it, expect-it) take a metadata map after the docstring. Adding :focus true to this map will cause only that test/suite to be run. Removing it will return to the normal behavior (run all tests).

(defdescribe my-test
  "fancy test"
  {:focus true}
  ...)

Editor Integration

The entry-points are at lazytest.repl: run-all-tests, run-tests, and run-test-var. The first runs all loaded test namespaces, the second runs the provided namespaces (either a single namespace or a collection of namespaces), and the third runs a single test var. If your editor can define custom repl functions, then it's fairly easy to set these as your test runner.

Example configuration

Neovim with Conjure:

-- in your init.lua
local runners = require("conjure.client.clojure.nrepl.action")
runners["test-runners"].lazytest = {
  ["namespace"] = "lazytest.repl",
  ["all-fn"] = "run-all-tests",
  ["ns-fn"] = "run-tests",
  ["single-fn"] = "run-test-var",
  ["default-call-suffix"] = "",
  ["name-prefix"] = "#'",
  ["name-suffix"] = ""
}
vim.g["conjure#client#clojure#nrepl#test#runner"] = "lazytest"

VSCode with Calva:

"calva.customREPLCommandSnippets": [
    {
        "name": "Lazytest: Test All Tests",
        "snippet": "(lazytest.repl/run-all-tests)"
    },
    {
        "name": "Lazytest: Test Current Namespace",
        "snippet": "(lazytest.repl/run-tests $editor-ns)"
    },
    {
        "name": "Lazytest: Test Current Var",
        "snippet": "(lazytest.repl/run-test-var #'$top-level-defined-symbol)"
    }
],

IntelliJ with Cursive:

Name: Lazytest - Test All Tests
Execute Command: (lazytest.repl/run-all-tests)
Execution Namespace: Execute in current file namespace
Results: Print results to REPL output

Name: Lazytest - Test Current Namespace
Execute Command: (lazytest.repl/run-tests ~file-namespace)
Execution Namespace: Execute in current file namespace
Results: Print results to REPL output

Name: Lazytest - Test Current Var
Execute Command: (lazytest.repl/run-test-var #'~current-var)
Execution Namespace: Execute in current file namespace
Results: Print results to REPL output

Lazytest Internals

The smallest unit of testing is a test case, which is a function (see lazytest.test-case/test-case). When the function is called, it may throw an exception to indicate failure. If it does not throw an exception, it is assumed to have passed. The return value of a test case is always ignored. Running a test case may have side effects.

NOTE: The macros lazytest.describe/it and lazytest.describe/expect-it create test cases.

Tests cases are organized into suites. A test suite is a function (see lazytest.suite/suite) that returns a test sequence. A test sequence (see lazytest.suite/test-seq) is a sequence, possibly lazy, of test cases and/or test suites. Suites, therefore, may be nested inside other suites, but nothing may be nested inside a test case.

NOTE: The macros lazytest.describe/defdescribe and lazytest.describe/describe create test suites.

A test suite function SHOULD NOT have side effects; it is only used to generate test cases and/or other test suites.

A test runnner is responsible for expanding suites (see lazytest.suite/expand-suite) and running test cases (see lazytest.test-case/try-test-case). It may also provide feedback on the success of tests as they run.

The test runner also returns a sequence of results, which are either suite results (see lazytest.suite/suite-result) or test case results (see lazytest.test-case/test-case-result). That sequence of results is passed to a reporter, which formats results for display to the user. Multiple reporters are provided, see the namespace lazytest.reporters.

Making Emacs Indent Tests Properly

Put the following in .emacs:

(eval-after-load 'clojure-mode
  '(define-clojure-indent
     (defdescribe 'defun)
     (describe 'defun)
     (given 'defun)
     (expect-it 'defun)
     (it 'defun)))

License

Originally by Alessandra Sierra.

Currently developed by Noah Bogart.

Licensed under Eclipse Public License 1.0