Lazytest: A standalone test framework for Clojure

An alternative to clojure.test, aiming to be feature-rich and easily extensible.

Table of Contents

• Getting Started
• Why a new test framework?
• Supported dialects
• Usage
• Writing tests with Lazytest
• Partitioning Individual Tests and Suites
• Setup and Teardown
• Output
• Hooks (Plugins)
• Doc Tests
• Editor Integration
• Run Lifecycle Overview
• Lazytest Internals
• License

Getting Started

Add it to your deps.edn or project.clj:

{:aliases
 {:test {:extra-deps {io.github.noahtheduke/lazytest {:mvn/version "2.0.0"}}
         :extra-paths ["test"]
         :main-opts ["-m" "lazytest.main"]}}}

In a test file, import with:

(require '[lazytest.core :refer [defdescribe describe expect it]])

And then write a simple test:

(defdescribe seq-fns-test
  (describe keep
    (it "should reject nils"
      (expect (= '(1 2 3) (keep identity [nil 1 2 3]))))
    (it "should return a sequence"
      (expect (seq? (seq (keep identity [nil])))))))

From the command line:

$ clojure -M:test

  lazytest.readme-test
    seq-fns-test
      #'clojure.core/keep
        √ should reject nils
        × should return a sequence FAIL

lazytest.readme-test
  seq-fns-test
    #'clojure.core/keep
      should return a sequence:

Expectation failed
Expected: (seq? (seq (keep identity [nil])))
Actual: nil
Evaluated arguments:
 * ()

in lazytest/readme_test.clj:11

Ran 2 test cases in 0.00272 seconds.
1 failure.

Why a new test framework?

clojure.test has existed since 1.1 and while it's both ubiquitous and useful, it has a number of problems:

• is tightly couples running test code and reporting on it.
• are is strictly worse than doseq or mapv.
• clojure.test/report is ^:dynamic, but that leads to being unable to combine multiple reporters at once, or libraries such as Leiningen monkey-patching it.
• Tests can't be grouped or bundled in any meaningful way (no, defining test-ns-hook does not count).
• testing calls aren't real contexts, they're just strings.
• Fixtures serve a real purpose but because they're set on the namespace, their definition is side-effecting and using them is complicated and hard to reuse.

There exists very good libraries like Expectations v2, kaocha, eftest, Nubank's matcher-combinators, Cognitect's test-runner that improve on clojure.test, but they're all still built on a fairly shaky foundation. I think it's worthwhile to explore other ways of being, other ways of doing stuff. Is a library like lazytest good? is a testing framework like this good when used in clojure? I don't know, but I'm willing to try and find out.

Other alternatives such as Midje, classic Expectations, and speclj attempted to correct some of those issues and they made good progress. However, some (such as Midje) relied on non-list style (test => expected) and most don't worked well with modern repl-driven development practices (as seen by the popularity of the aforementioned clojure.test-compatible Expectations v2).

I like the ideas put forth in Alessandra's post above about Lazytest and hope to experiment with achieving them 14 years later, while borrowing heavily from the work in both the Clojure community and test runners frameworks in other languages.

Supported dialects

Lazytest supports Clojure (1.10+) and Babashka (v1.12.208+). It does not support Clojurescript or other flavors at this time.

Usage

With the above :test alias, call clojure -M:test [options] [path...] to run your test suite once, or clojure -M:test --watch [options] [path...] to use "Watch mode" (see below) to run repeatedly as files change. [path...] here means any file or directory. By default, Lazytest only runs tests found in the test/ directory, but these can be passed in as additional arguments or with the --dir flag (see below):

Any of the below [options] can also be provided:

• -d, --dir DIR: Directory containing tests. Can be given multiple times.
• -n, --namespace SYMBOL: Run only the specified test namespaces. Can be given multiple times.
• -v, --var SYMBOL: Run only the specified fully-qualified symbol. Can be given multiple times.
• -i, --include KEYWORD: Run only test sequences or vars with this metadata keyword. Can be given multiple times.
• -e, --exclude KEYWORD: Exclude test sequences or vars with this metadata keyword. Can be given multiple times.
• --output SYMBOL: Output format. Can be given multiple times. (Defaults to nested.)
• --hook SYMBOL: Load and include a hook in the run. (See below to learn more about hooks). Can be given multiple times.
• --md FILE: Run doc tests in markdown file. Can be given multiple times. (See Doc Tests below.)
• --watch: Runs under "Watch mode", which reloads and reruns your test suite as project or test code changes.
• --delay NUM: How many milliseconds to wait before checking for changes to reload. Only used in "Watch mode". (Defaults to 500.)
• --help: Print help information.
• --version: Print version information.

Watch mode

Watch mode uses clj-reload to reload all local changes on the classpath, plus any files containing namespaces that depend on the changed files. Watch mode defaults to lazytest.reporters/dots to make the output easier to read. By default, it checks for changes once every 500 milliseconds (1/2 a second), but this can be changed with --delay. Watch mode supports all of the other options as well, so you can select a different output style, specific directories, test namespaces, or test varsto check, etc.

Type CTRL-C to stop.

Writing tests with Lazytest

The primary api is found in lazytest.core namespace. It mimics the behavior-driven testing style popularized by libraries such as RSpec and Mocha.

Define tests with defdescribe, group test suites and test cases together into a suite with describe, and define test cases with it. describe can be nested. defdescribe's docstring is optional, describe and it's docstrings are not.

(defdescribe +-test "with integers"
  (it "computes the sum of 1 and 2"
    (expect (= 3 (+ 1 2))))
  (it "computes the sum of 3 and 4"
    (assert (= 7 (+ 3 4))))
  (describe "associative property"
    (it "works in both directions"
      (expect (= 3 (+ 1 2)))
      (expect (= 3 (+ 2 1))))))

The expect macro is like assert but carries more information about the failure, such as the given form, the returned value, and the location of the call. It throws an exception if the expression does not evaluate to logical true.

If an it runs to completion without throwing, the test case is considered to have passed.

The describe macro creates a test suite, a map containing :children (among other things) which are nested suites or test cases (created with it). It can be passed to other functions or tests, it can be updated with other code, it can be removed from parent suites. Unlike clojure.test/testing, it is not merely setting a context string that is used to generate helpful error messages.

Aliases

To help write meaningful tests, a couple aliases have been defined for those who prefer different vocabulary:

• context for describe (this is discouraged because it clashes with the :context block, but it's retained for consistency).
• specify for it
• should for expect

These can be used interchangeably:

(require '[lazytest.core :refer [context specify should]])

(defdescribe context-test
  (context "with integers"
    (specify "that sums work"
      (should (= 7 (+ 3 4)) "follows basic math")
      (expect (not= 7 (+ 1 1))))))

There are a number of experimental namespaces that define other aliases, with distinct behavior, if the base set of vars don't fit your needs:

• lazytest.experimental.interfaces.clojure-test to mimic clojure.test.
• lazytest.experimental.interfaces.midje to mimic Midje.
• lazytest.experimental.interfaces.qunit to mimic QUnit.
• lazytest.experimental.interfaces.xunit to mimic a standard xUnit framework.

Var Metadata

In addition to finding the tests defined with defdescribe, Lazytest also checks all vars for :lazytest/test metadata. If the :lazytest/test metadata is a function, a test case, or a test suite, it's treated as a top-level defdescribe for the associated var and executed just like other tests. :lazytest/test functions are given the doc string "`:lazytest/test` metadata".

How to write them:

(defn fn-example
  {:lazytest/test #(expect (= 1 1))}
  [])
(defn test-case-example
  {:lazytest/test (it "test case example docstring" (expect (= 1 1)))}
  [])
(defn describe-example
  {:lazytest/test
    (describe "top level docstring"
      (it "first test case" (expect (= 1 1)))
      (it "second test case" (expect (= 1 1))))}
  [])

How they're printed:

  lazytest.readme-test
    #'lazytest.readme-test/fn-example
      √ `:lazytest/test` metadata
    #'lazytest.readme-test/test-case-example
      √ test case example docstring
    #'lazytest.readme-test/describe-example
      top level docstring
        √ first test case
        √ second test case

These can get unweildy if multiple test cases are included before a given implementation, so I recommend either moving them to a dedicated test file or moving the attr-map to the end of the function definition:

(defn post-attr-example
  ([a b]
   (+ a b))
  {:lazytest/test
   (describe "Should be simple addition"
     (it "handles ints"
       (expect (= 2 (post-attr-example 1 1))))
     (it "handles floats"
       (expect (= 2.0 (post-attr-example 1.0 1.0)))))})

Partitioning Individual Tests and Suites

All of the test suite and test case macros (defdescribe, describe, it, expect-it) take a metadata map after the docstring. Adding :focus true to this map will cause only that test/suite to be run. Removing it will return to the normal behavior (run all tests).

(defdescribe focus-test
  (it "will be run"
    {:focus true}
    (expect (= 1 2)))
  (it "will be skipped"
    (expect (= 1 1))))

And adding :skip true to the metadata map will cause that test/suite to be not run:

(defdescribe skip-test
  (it "will be skipped"
    {:skip true}
    (expect (= 1 2)))
  (it "will be run"
    (expect (= 1 1))))

Additionally, you can use the cli option -n/--namespace to specify one or more namespaces to focus wholly, or you can use the cli option -v/--var to specify one or more fully-qualified vars to focus. This allows for testing from the command line without modifying source files.

To partition your test suite based on metadata, you can use -i/--include to only run tests with the given metadata, or -e/--exclude to skip tests with the given metadata.

Setup and Teardown

To handle set up and tear down of stateful architecture, Lazytest provides the context macros before, before-each, after-each, after, around, and around-each, along with the helper function set-ns-context!. You can call them directly in a describe block or add them to a :context vector in suite metadata, or you can write the function directly as a map with the macro names as keywords. (To read a more specific description of how this works, please read the section titled Run Lifecycle Overview.)

(require '[lazytest.core :refer [expect-it before before-each after-each after around]])

(defdescribe before-and-after-test
  (let [state (volatile! [])]
    (describe "before and after example"
      (before (vswap! state conj :before))
      (after (vswap! state conj :after))
      (expect-it "can do side effects" (vswap! state conj :expect)))
    (describe "results"
      (expect-it "has been properly tracked"
        (= [:before :expect :after] @state)))))

(defdescribe around-test
  (let [state (volatile! [])]
    (describe "around example"
      {:context [(around [f]
                   (vswap! state conj :around-before)
                   (f)
                   (vswap! state conj :around-after))]}
      (expect-it "can do side effects" (vswap! state conj :expect)))
    (describe "results"
      (expect-it "correctly ran the whole thing"
        (= [:around-before :expect :around-after] @state)))))

(defdescribe each-test
  (let [state (volatile! [])]
    (describe "each examples"
      {:context [{:before (fn [] (vswap! state conj :before))
                  :before-each (fn [] (vswap! state conj :before-each))}]}
      (expect-it "can do side effects" (vswap! state conj :expect-1))
      (expect-it "can do side effects" (vswap! state conj :expect-2)))
    (expect-it "has been properly tracked"
      (= [:before :before-each :expect-1 :before-each :expect-2] @state))))

Context functions run in two directions

Every around, before, around-each, and before-each function is called in the order its defined, and every after and after-each is called in the opposite order it's defined. This is because before/-each and after/-each are intended to act like one half of an around or around-each call, which are designed like clojure.core/with-open and other similar functions. So before is called forward, and then after is called backward. This can be confusing, but I promise it's worthwhile.

Context functions work in two different modes

Context functions that run once

The around/before/after context functions are run only by the suite or test case they're defined for. So a before at the top of a test var will be run once before any child is evaluated, and a nested around will only wrap the evaluation of any children suites or test-cases, not parent or sibling suites. The same is true for test-cases: Any around/before/after context functions will be evaluated only once for that specific test-case.

Context functions that run multiple times

The around-each/before-each/after-each context functions are not run for the suite they're defined for, they're run by every nested test case, no matter how nested. A given test case gathers all parent *-each context functions, and then executes them with around-each wrapping any before-each or after-each functions. As with after, after-each is evaluated in reverse declaration order.

Namespace-level context functions

To set context functions for an entire namespace, use set-ns-context!. There is currently no way to define run-wide context functions.

In clojure.test, (use-fixtures :each ...) will set the provided fixtures to wrap each test var. To achieve the same in Lazytest, define a var of the target context function and add it to the defdescribe's :context block of each var in the namespace. This is necessarily more tedious than use-fixtures, but it is also more explicit and gracefully handles special cases (define multiple functions to handle subtle differences, use whichever is situationally helpful).

(defonce ^:dynamic *db-connection* nil)
(def prep-db
  (around [f]
    (binding [*db-connection* (get-db-connection ...)]
      (f))))

(defdescribe needs-a-db-test
  {:context [prep-db]}
  (it "has the right connection"
    (expect (= 1 (count (sql/query *db-connection* "SELECT * FROM users;"))))))

Common Patterns

To make this very clear, here are some patterns I've seen in clojure.test test suites, and how they might look in Lazytest.

with-redefs to stub a logger

(ns cool.example-test
  (:require
    [clojure.test :refer [deftest testing is]]
    [clojure.tools.logging :as log]
    [cool.example :as c.e]))

(deftest cool-func-test
  (with-redefs [log/log* (constantly nil)]
    (testing "with keywords"
      (is (c.e/cool-func :a :b :c)))
    (testing "with strings"
      (is (c.e/cool-func "a" "b" "c")))))

(ns cool.example-test
  (:require
    [lazytest.core :refer [defdescribe describe around expect-it]]
    [clojure.tools.logging :as log]
    [cool.example :as c.e]))

(defdescribe cool-func-test
  (around [f]
    (with-redefs [log/log* (constantly nil)]
      (f))
  (describe "with keywords"
    (expect-it "works" (c.e/cool-func :a :b :c)))
  (describe "with strings"
    (expect-it "works" (c.e/cool-func "a" "b" "c")))))

use-fixtures :each to reset a dynamic variable for every test

(ns cool.example-test
  (:require
    [clojure.test :refer [deftest testing is use-fixtures]]
    [com.stuartsierra.component :as component]
    [cool.example :as c.e]))

(defn set-state-fn [f]
  (binding [c.e/*state* (component/start (c.e/new-system))]
    (f)))

(use-fixtures :each #'set-state-fn)

(deftest system-func-test
  (testing "example 1"
    (is (c.e/system-func *state* :foo :bar))))

(deftest system-func-2-test
  (testing "example 2"
    (is (c.e/system-func-2 *state* :foo :bar))))

(ns cool.example-test
  (:require
    [lazytest.core :refer [defdescribe describe around expect-it]]
    [com.stuartsierra.component :as component]
    [cool.example :as c.e]))

(def set-state-fn
  (around [f]
    (binding [c.e/*state* (component/start (c.e/new-system))]
      (f))))

(defdescribe system-func-test
  {:context [set-state-fn]}
  (describe "example 1"
    (expect-it "works" (c.e/system-func *state* :foo :bar))))

(defdescribe system-func-2-test
  {:context [set-state-fn]}
  (describe "example 2"
    (expect-it "works" (c.e/system-func-2 *state* :foo :bar))))

use-fixtures :once to share a database connection across all tests

(ns cool.example-test
  (:require
    [clojure.test :refer [deftest testing is use-fixtures]]
    [next.jdbc :as jdbc]
    [cool.example :as c.e]))

(defn set-db-connection [f]
  (with-open [c.e/*connection* (jdbc/get-connection c.e/datasource)
    (f)))

(use-fixtures :once #'set-db-connection)

(deftest system-func-test
  (testing "example 1"
    (is (c.e/system-func c.e/*connection* :foo :bar))))

(deftest system-func-2-test
  (testing "example 2"
    (is (c.e/system-func-2 c.e/*connection* :foo :bar))))

(ns cool.example-test
  (:require
    [lazytest.core :refer [defdescribe describe around expect-it set-ns-context!]]
    [next.jdbc :as jdbc]
    [cool.example :as c.e]))

(set-ns-context!
 [(around [f]
    (with-open [c.e/*connection* (jdbc/get-connection c.e/datasource)
      (f)))])

(defdescribe system-func-test
  (describe "example 1"
    (expect-it "works" (c.e/system-func c.e/*connection* :foo :bar))))

(defdescribe system-func-2-test
  (describe "example 2"
    (expect-it "works" (c.e/system-func-2 c.e/*connection* :foo :bar))))

Generating data to be used across a whole test

(ns cool.example-test
  (:require
    [clojure.test :refer [deftest testing is]]
    [clojure.tools.logging :as log]
    [cool.example :as c.e]))

(deftest state-func-test
  (let [state (c.e/generate-big-state)]
    (testing "with keywords"
      (is (c.e/state-func state :foo)))
    (testing "with strings"
      (is (c.e/state-func state "bar")))))

(ns cool.example-test
  (:require
    [lazytest.core :refer [defdescribe describe around expect-it]]
    [clojure.tools.logging :as log]
    [cool.example :as c.e]))

(defdescribe state-func-test
  (let [state (delay (c.e/generate-big-state))]
    (describe "with keywords"
      (expect-it "works" (c.e/state-func @state :foo)))
    (describe "with strings"
      (expect-it "works" (c.e/state-func @state "bar")))))

Output

Lazytest comes with a number of reporters built-in. These print various information about the test run, both as it happens and surrounding execution. They are specified at the cli with --output and can be simple symbols or fully-qualified symbols. If a custom reporter is provided, it must be fully-qualified. (Otherwise, Lazytest will try to resolve it to the lazytest.reporters namespace and throw an exception.)

Built-in Reporters

lazytest.reporters/nested

The default Lazytest reporter. Inspired heavily by Mocha's Spec reporter, it prints each suite and test case indented as they are written in the test files.

  lazytest.core-test
    it-test
      √ will early exit
      √ arbitrary code
    with-redefs-test
      redefs inside 'it' blocks
        × should be rebound FAIL
      redefs outside 'it' blocks
        √ should not be rebound

lazytest.core-test
  with-redefs-test
    redefs inside 'it' blocks
      should be rebound:

this should be true
Expected: (= 7 (plus 2 3))
Actual: false
Evaluated arguments:
 * 7
 * 6
Only in first argument:
7
Only in second argument:
6

in lazytest/core_test.clj:29

Ran 90 test cases in 0.06548 seconds.
1 failure.

lazytest.reporters/dots

A minimalist reporter. Prints passing test cases as green . and failures as red F during the test run. Test suites are grouped with parentheses ((/)). It also prints the failure results and summary as in lazytest.reporters/nested, which has been elided below for brevity.

(...)(..F................)(.....)(..)(..)(....)(........)(........................................)(.......)

lazytest.reporters/clojure-test

Mimics clojure.test's default reporter, treating suite and test-case docstrings as testing strings.

Testing lazytest.core-test

FAIL in (with-redefs-test) (lazytest/core_test.clj:29)
with-redefs-test redefs inside 'it' blocks should be rebound
this should be true
expected: (= 7 (plus 2 3))
  actual: false

Ran 25 tests containing 90 test cases.
1 failure, 0 errors.

lazytest.reporters/quiet

Prints nothing. Useful if all you want is the return code.

lazytest.reporters/debug

Prints loudly about every step of the run. Incredibly noise, not recommended for anything other than debugging Lazytest internals.

Writing Custom Reporters

The requirements for writing a custom reporter is fairly simple: It needs to be an ifn that accepts 2 arguments, as it will be called by the runner on the run's config and the suite or test-case or result at that moment.

All of the built-in reporters are multimethods, as that provides the easiest means of handling each type object to be reported on. The objects are maps that have appropriate :type entries, and the public var lazytest.reporters/reporter-dispatch exists to make this easy: (defmulti nested* {:arglists '([config m])} #'reporter-dispatch). Each step in the run generally has a pair of types, one before the current suite/case is run and one after, with an additional :results holding the nested results from the children or the test case's result.

Here is a list of the types, along with info about when they are called and what the object might contain:

• :begin-test-run, :end-test-run: The top-level suite, parent to all test suites and cases that will be executed this run.
• :begin-test-ns, :end-test-ns: A suite for a given namespace, typically parent to all of the test vars in that namespace.
• :begin-test-var, :end-test-var: A suite for a single test var (typically from a defdescribe).
• :begin-test-suite, :end-test-suite: A stand-alone suite (typically from a describe).
• :begin-test-case, :end-test-case: A single test case (typically from an it), before it has been run.

Additionally, the test case's result will be reported between :begin-test-case and :end-test-case:

• :pass: The test case passed successfully.
• :fail: The test case failed for some reason.

Hooks (Plugins)

Lazytest supports writing plugins, called hooks, which can modify the state of a run while it is being executed. The authors of Lazytest can't predict every need, so we've added hooks as a means of "hooking" into the runtime system. Hooks are functions (generally multimethods) that dispatch on a keyword to do different things (set custom data, print info, change something about the data).

Using Existing Hooks

Hooks can be used with the --hook option, which can be given multiple times. For example, clojure -M:dev:test:lazytest --hook custom.ns/foo --hook custom.ns/bar will load and include both custom.ns/foo and custom.ns/bar as hooks during the run. Like reporters, if the specified symbol is not fully-qualified, it will be assumed to exist in lazytest.hooks, which is where the built-in hooks live.

Each hook can provide its own cli options, so it can be helpful to call clojure ... --hook custom.ns/foo --help to see what options are made available by the included hooks.

Built-in Hooks

The two built-in hooks run by default but they can be included and disabled by passing in the right flag. This is a deliberate design choice and not every hook will work like this.

lazytest.hooks/profiling

Print the slowest namespaces and test vars by duration. By default, it will print 5 namespaces and 5 vars, but this can be changed with --profiling-count. Can be disabled with --no-profiling.

$ clojure -M:dev:test:lazytest --hook profiling

Top 5 slowest test namespaces (0.48764 seconds, 45.9% of total time)
  lazytest.libs-test 0.40812 seconds
  lazytest.main-test 0.03445 seconds
  lazytest.find-test 0.01646 seconds
  lazytest.reporters-test 0.01585 seconds
  lazytest.core-test 0.01277 seconds

Top 5 slowest test vars (0.47278 seconds, 44.5% of total time)
  lazytest.libs-test/honeysql-test 0.40808 seconds
  lazytest.main-test/filter-ns-test 0.03442 seconds
  lazytest.find-test/find-var-test-value-test 0.01643 seconds
  lazytest.reporters-test/results-test 0.00791 seconds
  lazytest.core-test/expect-helpers-test 0.00594 seconds

lazytest.hooks/randomize

Randomize the order of namespaces and suites in namespaces during a test run. By default, it will randomize all namespaces, all test vars with namespaces, and then all nested suites and test-cases. This does not shuffle test vars between namespaces nor does it move nested suites or test cases around; this is merely a re-ordering of each child object.

The granularity can be changed with --randomize: all for everything (default), ns for only shuffling namespapces, var for only shuffling test vars, suites for only shuffling the suites within test vars, and none to disable.

Likewise, the shuffle is done with a random seed which is printed at the end of the run. The same ordering can be achieved by passing --randomize-seed with a previous run's seed.

$ clojure -M:dev:test:lazytest --hook randomize

lazytest.order-test
  One
    √ 1 equals one
  Four
    √ 4 equals four
    √ 1 equals one
    √ 5 equals five
    √ 3 equals three
    √ 2 equals two
  Two
    √ 2 equals two
  Three
    √ 3 equals three

Ran with --seed 263867813

Writing Custom Hooks

defhook

The primary means of writing a custom hooks is the helper macro lazytest.hooks/defhook. It creates a multimethod with the built-in hook dispatch function, defines a no-op :default, and has syntax similar to extend-protocol. The available hook methods are defined by the hook keywords, listed below. Each takes a config map and a source object. (Unlike manually created hook functions, the hook-type argument is inserted for you.)

(require '[lazytest.hooks :refer [defhook]])

(defhook yell
  "Prints a message at the start and end of the whole run."
  (cli-opts [config opts]
    (into opts
      [[nil "--[no-]yell" "Yell loudly"
        :id :yell/enabled
        :default true]]))
  (pre-test-run
    [config m]
    (println "STARTING")
    m)
  (post-test-run
    [config m]
    (println "ENDING")
    m))

(with-out-str (yell nil {:random :object} :post-test-run))
;; => "ENDING\n"

The current set of hook keywords, along with relevant details:

• :cli-opts: Called before config is built or cli opts have been fully parsed. Input is a vector of cli options.
• :config: Called after reporters and hooks have been resolved. Input is the config map.
• :pre-test-run: Called before the runner has started a full run. Won't be called if lazytest.runner/run-test-var or lazytest.repl/run-test-var is used. Input is the entire run's suite.
• :post-test-run: Called after the runner has finished a full run. Won't be called if lazytest.runner/run-test-var or lazytest.repl/run-test-var is used. Input is a suite-result, with the original suite stored under :source.
• :pre-test-suite: Called for every suite (namespace, var, nested suites). Input is the suite.
• :post-test-suite: Called after each suite (namespace, var, nested suites) has run. Input is a suite-result, with the original suite stored under :source.
• :pre-test-case: Called for every test-case that is executed. Input is the test-case.
• :post-test-case: Called after each test-case is executed. Input is the test-case-result, with the original test-case stored under :source.

Manually writing a hook function

Hooks are merely functions that are called with the same 3 arguments: the config map, a source object, and the hook-type keyword . The hook-type keyword is used as the dispatch value, and will be limited to the existing set of hook keywords (as listed above). It is expected that a given hook function will do different things based on hook-type, but that is not always the case.

The hook function should return nil (indicating a no-op) or the source object, modified as desired.

(defmulti example-hook-mm {:arglists '([config m hook-type])} #'lazytest.hooks/hook-dispatch)
(defmethod example-hook-mm :default [config m _] m)
(defmethod example-hook-mm :config [config config _] ...)
(defmethod example-hook-mm :pre-test-run [config suite _] ...)

(defn example-hook-func [obj config hook-type]
  (case hook-type
    :config ...
    :pre-test-run ...
    #_:else nil))

Doc Tests

Lazytest can run tests in code blocks of your markdown files with --md FILE. It looks for any triple backtic-delimited code block that has clojure or clj as the language specifier, and that doesn't have lazytest/skip=true in the info-string, bundles it into a standalone describe block, and then runs all of the suites as a single suite under the name of the markdown file.

It determines what should be considered a test ((expect (= x y))) by the presence of =>. Code immediately before a line containing => (leading ; optional) is treated as the actual, and the value after treated as the expected result.

This will run:

```clojure
(defn adder [a b]
  (+ a b))

(adder 5 6)
;; => 11
```

Whereas these will not (first is skipped, second isn't "clojure" or "clj"):

```clojure lazytest/skip=true
(System/exit 1)
;; => exit!!!
```

```clojurescript
print("Hello world!")
```

Additionally, a custom string can be used instead of the default (headers from the markdown file) by using the info-string lazytest/describe:

```clojure lazytest/describe=easy-adder
(+ 5 6)
;; => 11
```

will be printed as:

  readme-md
    easy-adder
      √ Doc Tests

Editor Integration

The entry-points are at lazytest.repl: run-all-tests, run-tests, and run-test-var. The first runs all loaded test namespaces, the second runs the provided namespaces (either a single namespace or a collection of namespaces), and the third runs a single test var. If your editor can define custom repl functions, then it's fairly easy to set these as your test runner.

Neovim

Neovim with Conjure:

-- in your init.lua
local runners = require("conjure.client.clojure.nrepl.action")
runners["test-runners"].lazytest = {
  ["namespace"] = "lazytest.repl",
  ["all-fn"] = "run-all-tests",
  ["ns-fn"] = "run-tests",
  ["single-fn"] = "run-test-var",
  ["default-call-suffix"] = "",
  ["name-prefix"] = "#'",
  ["name-suffix"] = ""
}
vim.g["conjure#client#clojure#nrepl#test#runner"] = "lazytest"

VSCode

VSCode with Calva:

"calva.customREPLCommandSnippets": [
    {
        "name": "Lazytest: Test All Tests",
        "snippet": "(lazytest.repl/run-all-tests)"
    },
    {
        "name": "Lazytest: Test Current Namespace",
        "snippet": "(lazytest.repl/run-tests $editor-ns)"
    },
    {
        "name": "Lazytest: Test Current Var",
        "snippet": "(lazytest.repl/run-test-var #'$top-level-defined-symbol)"
    }
],

IntelliJ

IntelliJ with Cursive:

Name: Lazytest - Test All Tests
Execute Command: (lazytest.repl/run-all-tests)
Execution Namespace: Execute in current file namespace
Results: Print results to REPL output

Name: Lazytest - Test Current Namespace
Execute Command: (lazytest.repl/run-tests ~file-namespace)
Execution Namespace: Execute in current file namespace
Results: Print results to REPL output

Name: Lazytest - Test Current Var
Execute Command: (lazytest.repl/run-test-var #'~current-var)
Execution Namespace: Execute in current file namespace
Results: Print results to REPL output

Run Lifecycle Overview

This is inspired by Mocha's excellent documentation.

From the CLI

1. A user runs Lazytest, either through leiningen or Clojure CLI.
2. Lazytest parses the command line arguments to determine the relevant configuration.
3. Lazytest finds test files. If the user provides --dir, then every file in the file trees of all given directories are checked. Otherwise, all files within the test directorie are checked.
4. Lazytest loads all test files. Using tools.namespace, the namespace of each .clj is extracted and required, which creates the necessary vars.
5. Lazytest gathers all test vars from the required namespaces. It checks each var in each namespace against the following list of questions.
   1. Is the var defined with defdescribe? Call the defdescribe-constructed function and use the result.
   2. Does the var point to a suite? Resolve the var and use the result.
   3. Does the var have :lazytest/test metadata that is either a suite (describe) or a test case (it)? Create a new suite with describe and set the :lazytest/test metadata as a child.
   4. Does the var have :lazytest/test metadata that is a function? Create a new suite with describe, create a new test case with it, and then set the docstring for the test case to :lazytest/test metadata, and the body to calling the :lazytest/test metadata function.
6. Lazytest groups each namespace into a :lazytest/ns suite, and then groups all of the namespace suites into a :lazytest/run suite.
7. Lazytest does a depth-first walk of the run suite, filtering nses by --namespace, vars by --var, and all suites and test cases by --include or --exclude (with :focus being automatically included). These are prioritized as such:
   1. --namespace narrows all namespaces to those that exactly match. The namespaces of --var vars are included as well. If --namespace is not provided, all namespaces are selected.
   2. --var narrows all vars from the selected namespaces. If --namespace is provided, all vars from those namespaces are selected as well. If --var is not provided, all vars are selected.
   3. The suite for each var is selected by selecting all --include or :focus metadata suites and tests cases and then removing all --exclude suites and test cases. If no suites or test cases have :focus metadata or --include hasn't been provided, then everything is selected. (To be clear, --exclude overrides :focus and --include.)
8. Lazytest calls the runner on the filtered run suite.
   • For suites:
     1. If there are any around context functions, combine them with clojure.test/join-fixtures, and then execute the rest of the steps in a thunk wrapped in the combined around function.
     2. Run each before context function.
     3. For each child in :children, restart from step 1 of the appropriate sequence.
     4. Run each after context function.
   • For test cases:
     1. If there are any around context functions, combine them with clojure.test/join-fixtures, and then execute the rest of the steps in a thunk wrapped in the combined around function.
     2. Run each before context function.
     3. Run each before-each function (including from all parents), outermost first, in definition order.
     4. Execute the test function, get the test-case-result.
     5. Run each after-each function (including from all parents), innermost first, in definition order.
     6. Run each after context function.
9. Depending on the chosen reporter, Lazytest prints the results of each suite and test case immediately or at another point.
10. The run is ended with System/exit, and the exit value is either 0 for no failures or 1 for any number of failures.

Programmatically

The process is roughly the same as from the CLI, but with CLI-specific steps skipped.

1. Build a suite.
   • If using lazytest.repl/run-tests, the specified namespace used as the required namespace.
   • If using lazytest.repl/run-all-tests, all currently loaded are used (found with clojure.core/all-ns).
   • If using lazytest.repl/run-test-var, the single var is used as the suite.
2. If not given a var, step 5 is executed as described above to produce a suite.
3. Steps 7-9 are executed as described above on the suite, with the note that only :focus is considered when filtering.
4. The results from the run are summarized and returned to the caller.

Lazytest Internals

The smallest unit of testing is a test case (see lazytest.test-case/test-case). When the :body function is called, it may throw an exception to indicate failure. If it does not throw an exception, it is assumed to have passed. The return value of a test case is always ignored. Running a test case may have side effects.

Tests cases are organized into suites (see lazytest.suite/suite). A suite has :children, which is a sequence, possibly lazy, of test cases and/or test suites. Suites, therefore, may be nested inside other suites, but nothing may be nested inside a test case.

A test suite body SHOULD NOT have side effects; it is only used to generate test cases and/or other test suites.

The test runner is responsible for gathering suites (see lazytest.find/find-suite and lazytest.filter/filter-tree) and running test cases (see lazytest.test-case/try-test-case). It may also provide feedback on the success of tests as they run.

The test runner also returns a sequence of results, which are either suite results (see lazytest.suite/suite-result) or test case results (see lazytest.test-case/test-case-result). That sequence of results is passed to a reporter, which formats results for display to the user. Multiple reporters are provided, see the namespace lazytest.reporters.

License

Originally by Alessandra Sierra.

Currently developed by Noah Bogart.

Licensed under Eclipse Public License 1.0