kaocha.report — lambdaisland/kaocha 0.0-409

Reporters generate textual output during a test run, providing real-time information on test progress, failures, errors, and so forth. They are in nature imperative and side-effectful, they generate output on an output stream (typically stdout), based on test events. Some reporters are also used to track state. This is unfortunate as it goes against Kaocha's functional design, but since we want test runs to be interruptible it is somewhat inevitable.

The concept of reporters is directly taken from clojure.test, but is used in Kaocha also when running other types of tests.

A reporter is a function which takes a single argument, a map. The map will have a :type key indicating the type of event, e.g. :begin-test-var, :fail, :pass, or :summary.

Reporters as imagined in clojure.test are a flawed design, we try to make the best of it. See also the monkeypatching of clojure.test/do-test in kaocha.monkey-patch, which is necessary to be able to intercept failures quickly in case the users runs with --fail-fast enabled. The patch also ensures that the current testable is always available in the event map under :kaocha/testable,

Kaocha differs from stock clojure.test in that multiple reporters can be active at the same time. On the command line you can specify --reporter multiple times, in tests.edn you can pass a vector to :kaocha/reporter, and/or point at a var which itself defines a vector of functions. Each of the given functions will be called in turn for each event generated.

This has allowed Kaocha to split the functionality of reporters up, making them more modular. E.g. kaocha.report/report-counters only keeps the fail/error/pass/test counters, without concerning itself with output, making it reusable.

This namespace implements the reporters provided by Kaocha out of the box that don't need extra dependencies. Others like e.g. the progress bar are in their own namespace to prevent loading files we don't need, and thus slowing down startup.

Issues with clojure.test reporters

clojure.test provides reporters as a way to extend the library. By default clojure.test/report is a multimethod which dispatches on :type, and so libraries can extend this multimethod to add support for their own event types. A good example is the :mismatch event generated by matcher-combinators.

Tools can also rebind clojure.test/report, and use it as an interface for capturing test run information.

The problem is that these two approaches don't mesh. When tools (like Kaocha, CIDER, Cursive, etc.) rebind clojure.test/report, then any custom extensions to the multimethod disappear.

This can also cause troubles when a library which extends clojure.test/report gets loaded after it has been rebound. This was an issue for a while in test.check, which assumed report would always be a multimethod (this has been rectified). For this reasons Kaocha only rebinds report after the "load" step.

Kaocha tries to work around these issues to some extent by forwarding any keys it does not know about to the original clojure.test/report multimethod. This isn't ideal, as these extensions are not aware of Kaocha's formatting and output handling, but it does provide some level of compatiblity with third party libraries.

For popular libraries we will include reporter implementations that handle these events in a way that makes sense within Kaocha, see e.g. kaocha.matcher-combinators. Alternatively library authors can themselves strive for Kaocha compatiblity, we try to give them the tools to enable this, through keyword derivation and custom multimethods.

Custom event types

kaocha.report makes use of Clojure's keyword hierarchy feature to determine the type of test events. To make Kaocha aware of your custom event, first add a derivation from :kaocha/known-type, this will stop the event from being propagated to the original clojure.test/report

(kaocha.hierarchy/derive! :mismatch :kaocha/known-key)

If the event signals an error or failure which causes the test to fail, then derive from :kaocha/fail-type. This will make Kaocha's existing reporters compatible with your custom event.

(kaocha.hierarchy/derive! :mismatch :kaocha/fail-type)

assertion-type clj

(assertion-type m)

Given a clojure.test event, return the first symbol in the expression inside (is).

clojure-test-report clj multimethod

debug clj

(debug m)

dispatch-extra-keys clj

(dispatch-extra-keys m)

Call the original clojure.test/report multimethod when dispatching an unknown key. This is to support libraries like nubank/matcher-combinators that extend clojure.test/assert-expr, as well as clojure.test/report, to signal special conditions.

doc clj multimethod

doc-print-contexts clj

(doc-print-contexts contexts & [suffix])

doc-printed-contexts clj

documentation clj

Reporter that prints an overview of all tests bein run using indentation.

dots clj

Reporter that prints progress as a sequence of dots and letters.

dots* clj multimethod

fail-fast clj

(fail-fast m)

Fail fast reporter, add this as a final reporter to interrupt testing as soon as a failure or error is encountered.

fail-summary clj multimethod

print-expr clj multimethod

print-output clj

(print-output m)

report-counters clj multimethod

result clj multimethod

tap clj

(tap {:keys [type] :as m})

Reporter for the TAP protocol (Test Anything Protocol).

testing-vars-str clj

(testing-vars-str {:keys [file line testing-vars kaocha/testable] :as m})

Returns a string representation of the current test. Renders names in :testing-vars as a list, then the source file and line of current assertion.