Reporters generate textual output during a test run, providing real-time information on test progress, failures, errors, and so forth. They are in nature imperative and side-effectful, they generate output on an output stream (typically stdout), based on test events. Some reporters are also used to track state. This is unfortunate as it goes against Kaocha's functional design, but since we want test runs to be interruptible it is somewhat inevitable.
The concept of reporters is directly taken from clojure.test, but is used in Kaocha also when running other types of tests.
A reporter is a function which takes a single argument, a map. The map will have a
:type key indicating the type of event, e.g.
Reporters as imagined in
clojure.test are a flawed design, we try to make the best of it. See also the monkeypatching of
kaocha.monkey-patch, which is necessary to be able to intercept failures quickly in case the users runs with
--fail-fast enabled. The patch also ensures that the current testable is always available in the event map under
Kaocha differs from stock
clojure.test in that multiple reporters can be active at the same time. On the command line you can specify
--reporter multiple times, in
tests.edn you can pass a vector to
:kaocha/reporter, and/or point at a var which itself defines a vector of functions. Each of the given functions will be called in turn for each event generated.
This has allowed Kaocha to split the functionality of reporters up, making them more modular. E.g.
kaocha.report/report-counters only keeps the fail/error/pass/test counters, without concerning itself with output, making it reusable.
This namespace implements the reporters provided by Kaocha out of the box that don't need extra dependencies. Others like e.g. the progress bar are in their own namespace to prevent loading files we don't need, and thus slowing down startup.
clojure.test provides reporters as a way to extend the library. By default
clojure.test/report is a multimethod which dispatches on
:type, and so libraries can extend this multimethod to add support for their own event types. A good example is the
:mismatch event generated by matcher-combinators.
Tools can also rebind
clojure.test/report, and use it as an interface for capturing test run information.
The problem is that these two approaches don't mesh. When tools (like Kaocha, CIDER, Cursive, etc.) rebind
clojure.test/report, then any custom extensions to the multimethod disappear.
This can also cause troubles when a library which extends
clojure.test/report gets loaded after it has been rebound. This was an issue for a while in test.check, which assumed
report would always be a multimethod (this has been rectified). For this reasons Kaocha only rebinds
report after the "load" step.
Kaocha tries to work around these issues to some extent by forwarding any keys it does not know about to the original
clojure.test/report multimethod. This isn't ideal, as these extensions are not aware of Kaocha's formatting and output handling, but it does provide some level of compatiblity with third party libraries.
For popular libraries we will include reporter implementations that handle these events in a way that makes sense within Kaocha, see e.g.
kaocha.matcher-combinators. Alternatively library authors can themselves strive for Kaocha compatiblity, we try to give them the tools to enable this, through keyword derivation and custom multimethods.
kaocha.report makes use of Clojure's keyword hierarchy feature to determine the type of test events. To make Kaocha aware of your custom event, first add a derivation from
:kaocha/known-type, this will stop the event from being propagated to the original
(kaocha.hierarchy/derive! :mismatch :kaocha/known-key)
If the event signals an error or failure which causes the test to fail, then derive from
:kaocha/fail-type. This will make Kaocha's existing reporters compatible with your custom event.
(kaocha.hierarchy/derive! :mismatch :kaocha/fail-type)