Kaocha is designed to be extensible and customizable, so that it can adapt to the needs of different projects, and so that it can act as a common base layer upon which innovative or advanced testing features can be delivered.
The key to understanding Kaocha is understanding the three steps that Kaocha goes through to perform a test run, and the three associated data structures.
A Kaocha test run consists of three part
In the configure step the configuration file is loaded, normalized, and any
command line options merged in. Plugins are loaded at this staged, and given a
chance to update the configuration. The result is a Kaocha configuration (spec:
:kaocha/config
)
After this each test suite loads. This means loading the test namespaces and
finding out which tests are in them. The loading specifics are delegated to the
test suite type. After the load step the Kaocha configuration has transformed
into a test plan (spec: :kaocha/test-plan
), containing a nested collection of
"testables", providing a detailed overview of which tests will be run.
Finally the test-plan gets run, which means recursively executing the collection
of testables. Each testable gets updated with information about the test run:
whether it failed/passed/errored, captured output or exceptions, etc. After this
step the test-plan has transformed into a test result. (spec: :kaocha/result
)
It's a good idea to keep the specs handy as a reference.
A testable is a map containing the testable's type, id, type specific information, and nested testables.
There are three versions of testables. In the configuration there are only
top-level testables, also called test suites, stored under :kaocha/tests
.
{:kaocha/tests [{:kaocha.testable/type :kaocha.type/clojure.test
:kaocha.testable/id :unit
:kaocha/source-paths ["src"]
:kaocha/test-paths ["test"]}]}
Loading these tests is done with the kaocha.testable/-load
multimethod, which
dispatches on the testable type.
After the load step these have become test-plan testables, with lots of extra info, including nested test-plan testables.
{:kaocha.test-plan/tests [{:kaocha.testable/type :kaocha.type/clojure.test
:kaocha.testable/id :unit
:kaocha/source-paths ["src"]
:kaocha/test-paths ["test"]
:kaocha.testable/meta {}
:kaocha.test-plan/tests [{:kaocha.testable/type :kaocha.type/ns
:kaocha.testable/id :kaocha.runner-test
,,,
:kaocha.test-plan/tests [{:kaocha.testable/type :kaocha.testable/var
:kaocha.testable/id :kaocha.runner-test/main-test
,,,}]}]}]}
Running these tests is again type specific, each type has an implementation of
kaocha.testable/-run
, which recursively calls kaocha.testable/run-tests
,
which collects the results into the test result data structure.
{:kaocha.result/tests [{:kaocha.testable/type :kaocha.type/clojure.test
:kaocha.testable/id :unit
:kaocha/source-paths ["src"]
:kaocha/test-paths ["test"]
:kaocha.testable/meta {}
:kaocha.result/tests [{:kaocha.testable/type :kaocha.type/ns
:kaocha.testable/id :kaocha.runner-test
,,,
:kaocha.result/tests [{:kaocha.testable/type :kaocha.testable/var
:kaocha.testable/id :kaocha.runner-test/main-test
:kaocha.result/count 1
:kaocha.result/pass 1
:kaocha.result/fail 0
:kaocha.result/error 0
,,,}]}]}]}
A plugin consists of functions that get run when certain "hooks" within Kaocha fire, bundled in a map from keyword to function.
To write a Kaocha plugin you implement the kaocha.plugin/-register
multimethod. This allows the plugin to add itself to the "plugin chain", a
vector of plugin maps.
(ns my.kaocha.plugin
(:require [kaocha.plugin :as p]))
(defmethod p/-register :my.kaocha/plugin [_name plugins]
(conj plugins
{:kaocha.hooks/config
(fn [config]
(assoc config ::setting :foo))
:kaocha.hooks/pre-run
(fn [test-plan]
(println "run is starting!")
test-plan)}))
Plugin names must be namespaced keywords. If your plugin is called
:foo.bar/baz
then it must be implemented in the namespace foo.bar
or
foo.bar.baz
. This will allow Kaocha to automatically load the plugin before
calling it.
To take the boilerplate out of writing plugins you are encouraged to use the
defplugin
macro.
These are all the hooks a plugin can implement. Note that each must return its first argument (possibly updated).
(ns my.kaocha.plugin
(:require [kaocha.plugin :as p]))
(p/defplugin my.kaocha/plugin
;; Install extra CLI options and flags.
(cli-options [opts]
opts)
;; Alter the configuration. Useful for setting default values.
(config [config]
config)
;; Runs before the load step
(pre-load [config]
config)
;; Runs after the load step
(post-load [test-plan]
test-plan)
;; Runs before the run step
(pre-run [test-plan]
test-plan)
;; Runs before each individual test
(pre-test [test test-plan]
test)
;; Runs after each individual test
(post-test [test test-plan]
test)
;; Runs after the run step
(post-run [result]
result)
;; Allows "wrapping" the run function
(wrap-run [run test-plan]
run)
;; Runs before the reporter
(pre-report [event]
event))
Start with the boilerplate, i.e. a namespace + empty defplugin declaration.
(ns my.kaocha.plugin
(:require [kaocha.plugin :refer [defplugin]]))
(defplugin my.kaocha/plugin
,,,)
From there you could already add it to tests.edn
and e.g. start in --watch
and start iterating, but that's a pretty coarse workflow. For more fine-grained
work you can use kaocha.repl
, in particular kaocha.repl/config
and
kaocha.repl/test-plan
(we should probably also add a kaocha.repl/result
so
you can inspect the final result data).
So say you have a pre-load
hook, and your plugin is enabled in tests.edn
,
then you can call (kaocha.repl/test-plan)
and see the effects of your plugin.
defplugin
will actually define several vars, plus the final defmethod
which
registers the plugin. So you can test your hooks in isolation.
(defplugin my.kaocha/plugin
"Docstring"
(cli-options [opts] opts)
(config [config] config)
(pre-load [config] config))
;; This defines
(defn plugin-cli-options-hook [opts] opts)
(defn plugin-config-hook [config] config)
(defn plugin-pre-load-hook [config] config)
(def plugin-hooks
{:kaocha.plugin/id :my.kaocha/plugin
:kaocha.plugin/description "Docstring"
:kaocha.hooks/cli-options plugin-cli-options-hook
:kaocha.hooks/config plugin-config-hook
:kaocha.hooks/pre-load plugin-pre-load-hook})
(defmethod kaocha.plugin/-register :my.kaocha/plugin [chain]
(conj chain plugin-hooks))
So this is great for unit tests (test the hooks directly), and should be helpful when developing from the REPL as well.
(my.kaocha.plugin/plugin-config-hook (kaocha.repl/config))
;; => ???
You may wonder why all this boilerplate, e.g. why does the -register
method
have to call conj
, on the plugin chain, instead of just returning the map with
hooks? The reason is this allows for plugins to do more complex things, like
injecting multiple plugins at once, adding a plugin before or after an other
one, or wrapping functions of other plugins.
Now of course the question is: which hooks to use and what to do with them. Generally your hooks will fall into two categories, either you're just using a hook to cause some side effect at a certain point in the execution, or you're manipulating Kaocha's data structures to change its behavior.
Kaocha is very data driven, so the idea is that e.g. by changing the config or
test-plan you can change its behavior. For instance you can implement special
test filtering with a pre-test
hook that does (assoc testable :kaocha.testable/skip true)
when a certain condition is met. Here you'll have
to poke around the source a bit, look for the place where you would normally
hack in your change, and then hope that there's a hook there and affordances to
cause the right behavior.
Final a general tip/best practice: if your plugin is in any way configurable,
then it should use the cli-options
and config
hooks, in such a way that
options specified on the CLI override those set in the config. The cli-options
hooks defines your command line flags, then in the config
hooks you can
inspect :kaocha/cli-options
in the config to find the flags used, and use them
to update the config, or provide a default. Any following hooks then look at the
config for the necessary settings (and so not directly at
:kaocha/cli-options
). You can look at the built-in plugins, most of them use
this pattern.
This is important because this way when a user uses --print-config
they see
those default values added by plugins, which they can copy to tests.edn
and
tweak. (you should use namespaced keywords based on the name of your plugin.)
You should also check out kaocha.test-util/with-test-ctx
, this is useful to
isolate unit tests from Kaocha itself.
Kaocha is designed to be a universal tool, able to run any type of test suite your project chooses to use. To make this possible it provides a way to implement custom test suite types.
In the test configuration every suite has a type.
{:kaocha/tests [{:kaocha.testable/type :kaocha.type/clojure.test
:kaocha.testable/id :unit}]}
When Kaocha encounters this test suite it will first try to load the type, by
requiring either the kaocha.type
or kaocha.type.clojure.test
namespace.
It will then validate the suite configuration using the
:kaocha.type/clojure.test
spec, so a custom test suite implementation must
register a Clojure spec with the same name as the suite type.
Finally a test suite implements two multimethods, one that handles Kaocha's load stage, and one that handles the run stage.
Here's a skeleton example of a test suite.
(ns kaocha.type.clojure.test
(:require [clojure.spec.alpha :as s]
[kaocha.testable :as testable]
[kaocha.load :as load]
[clojure.test :as t]))
(defmethod testable/-load :kaocha.type/clojure.test [testable]
(assoc :kaocha.testable test-plan/tests (load-tests ...)))
(defmethod testable/-run :kaocha.type/clojure.test [testable test-plan]
(t/do-report {:type :begin-test-suite})
(let [results (testable/run-testables (:kaocha.test-plan/tests testable) test-plan)
testable (-> testable
(dissoc :kaocha.test-plan/tests)
(assoc :kaocha.result/tests results))]
(t/do-report {:type :end-test-suite
:kaocha/testable testable})
testable))
(s/def :kaocha.type/clojure.test (s/keys :req [:kaocha/source-paths
:kaocha/test-paths
:kaocha/ns-patterns]))
(hierarchy/derive! :kaocha.type/clojure.test :kaocha.testable.type/suite)
Start by thinking about the hierarchy of your test types. You typically have a top level "test suite" type, an intermediate "group" type, and the actual individual tests, called the "leaf" type. You can think of suite/group/leaf corresponding to directory/file/test, although it doesn't have to be that way.
For instance for clojure.test one or more directories for a suite, this suite
consists of namespaces, and each namespace contains test vars, so the hierarchy
is :kaocha.type/clojure.test
> :kaocha.type/ns
> :kaocha.type/var
.
For Cucumber tests the hierarchy is :kaocha.type/cucumber
>
:kaocha.cucumber-feature
> :kaocha.type/cucumber-scenario
.
You could have more or fewer levels. The top one is always known as the suite, the bottom one as the leaf, the intermediate ones as groups.
For each test type you implement kaocha.testable/-run
, and for the suite and
groups you implement kaocha.testable/-load
. Then you use
kaocha.testable/load-testables
/ kaocha.testable/run-testables
to perform
the recursion.
Use kaocha.hierarchy/derive!
to mark your test types as suite/group/leaf.
(hierarchy/derive! :kaocha.type/clojure.test :kaocha.testable.type/suite)
(hierarchy/derive! :kaocha.type/ns :kaocha.testable.type/group)
(hierarchy/derive! :kaocha.type/var :kaocha.testable.type/leaf)
When implementing -load
your job is to transform a configuration testable into
a test-plan testable, so you should should dissoc :kaocha/tests
and assoc :kaocha.test-plan/tests
.
-load
is responsible for adding the test directories to the classpath (if this
applies for your test type). The helpers in kaocha.load
will come in handy for
this.
If an error occurs while loading files then signal that by adding the
:kaocha.testable/load-error
to the testable, with as value the caught
exception. You'll deal with signaling the error to the user during the -run
step.
The -run
implementation generally starts by calling clojure.test/do-report
with a "begin" event, then it runs either the contained
:kaocha.test-plan/tests
(for a suite/group), or runs the actual test, and then
calls clojure.test/do-report
agin with an :end
event.
The return value from -run
is a :kaocha.result/testable
, which is like a
:kaocha.test-plan/testable
, but has (for a suite/group test)
:kaocha.result/tests
rather than :kaocha.test-plan/tests
, or has (for a leaf
test) result stats (count, error, fail, pass, pending) added.
To gather result stats you can use kaocha.type/with-report-counters
/
kaocha.type/report-count
.
The -run
method for a leaf test should also take care of wrapping the core
test logic in any wrapping functions provided by :kaocha.testable/wrap
on the
testable. This is important for output capturing to work correctly.
You should check that when your test fails, you get the right file and line
number in the output. Kaocha tries to detect this from the stacktrace, but that
doesn't always work. (see the kaocha.monkey-patch
namespace). Alternatively
bind kaocha.testable/*test-location*
to a map with :file
and :line
.
Before invoking the actual test logic, check for :kaocha.testable/load-error
,
and if it's there then signal a test error and finish. You can do this with the
kaocha.testable/handle-load-error
helper.
During the recursive invocations of -run
clojure.test
style events are
emitted by calling clojure.test/do-report
. Rather than reusing pre-existing
generic event types you should come up with event types that are specific to
your test type, then use kaocha.hierarchy/derive!
to attach semantics to them.
This is an example of event types, and the keywords they derive from.
:foo-test/begin-suite :kaocha/begin-suite
:foo-test/begin-ns :kaocha/begin-group
:foo-test/begin-test :kaocha/begin-test
:foo-test/assert-failed :kaocha/fail-type
:foo-test/precondition-failed :kaocha/fail-type
:foo-test/end-test :kaocha/end-test
:foo-test/begin-test :kaocha/begin-test
:pass :kaocha/known-key
:foo-test/end-test :kaocha/end-test
:foo-test/end-ns :kaocha/end-group
:foo-test/end-suite :kaocha/end-suite
Some notable parent types to inherit from
:kaocha/known-key
all events we emit should eventually inherit from
known-key. Any event we receive that is not a known-key will be propagated to
the original clojure.test/report
multimethod, for compatibility with
assertion libraries that emit their own custom events and extend the
multimethod to handle them.
:kaocha/fail-type
anything that fails the test should inherit from
fail-type
. Reporters that don't know about specific failure types can still
use this to do some reporting of failures, print captured output, etc.
:kaocha/deferred
events that inherit from deferred
are saved up during the
test run, and will be sent to clojure.test/report
during the summary step.
This is how we make sure that the details of test failures are only printed
all the way at the end during the :summary
step, rather than immediately as
they happen.
In general we try to use properly namespaced event types, but because Kaocha is
built on top of clojure.test (for better or for worse), we still use some of the
non-namespaced names used by clojure.test like :pass
, :fail
, :error
. The
main reason is to keep (some) compatibility with reporters written for
clojure.test
that are not Kaocha-aware. However there's no really good way to
do this, either we limit ourselves to the scope of clojure.test
's reporting
(which we don't want to do), or we go for a more semantically rich set of
events, but cause pre-existing reporters to misbehave.
We are well on the path to the latter, and so we will likely drop more of these non-namespaced ones in favor of kaocha-specific ones, and drop support for legacy reporters alltogether.
Reporters generate the test runners output. They are in their nature side-effectful, printing to stdout in response to events.
Before creating your own reporters, consider whether the same thing could be accomplished with plugins. They provide a more functional and composable interface, and should be preferred.
A reporter is a function which takes a single map as argument, with the map having a :type
key. Kaocha uses the same types as clojure.test
, but adds :begin-test-suite
and :end-test-suite
.
Kaocha contains fine-grained reporters, which you can combine, or mix with your own to get the desired output. A reporter can be either a function, or a sequence of reporters, which will all be called in turn. For instance, the default Kaocha reporter is defined as such:
(ns kaocha.report)
(def dots
"Reporter that prints progress as a sequence of dots and letters."
[dots* result])
Reporters intended for use with clojure.test
will typically call clojure.test/inc-report-counters
to keep track of stats. Reporters intended for use with Kaocha should not do this. Kaocha will always inject the kaocha.report.history/track
reporter which takes care of that.
A common use case for extending or replacing reporters is to support custom assertion functions which emit their own :type
of clojure.test
events.
(clojure.test/do-report {:type ::my-assertion, :message ..., :expected ..., :actual ...})
For this use case you might not have to implement a custom reporter at all.
First make Kaocha aware of your new event type. This will prevent it from being foward to the original clojure.test/report
, which by default just prints the map to stdout.
(kaocha.hierarchy/derive! ::my-assertion :kaocha/known-key)
If your event would cause a test to fail, then also mark it as a :kaocha/fail-type
(kaocha.hierarchy/derive! ::my-assertion :kaocha/fail-type)
This way Kaocha's built-in reporters will know that this event indicates a failure, and correctly report it in the test results. It will also cause a default failure message to be rendered based on the :message
, :expected
, and :actual
keys.
If you want to provide custom output then add an implementation of the kaocha.report/fail-summary
multimethod.
(defmulti kaocha.report/fail-summary ::my-assertion [m]
(println ...)
)
For a full example have a look at Kaocha's built-in matcher combinator support.
Built in reporters include
kaocha.report/dots
kaocha.report/documentation
kaocha.report.progress/report
Can you improve this documentation? These fine people already did:
Arne Brasseur, Whitt, Alex & Paulo Rafael FeodrippeEdit on GitHub
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close