Liking cljdoc? Tell your friends :D

Test Runners

Starting with version 0.2.15, the poly tool lets you choose your test runners.

Starting with version 0.2.19, you need at least 0.8.4 of the Kaocha test runner, and 0.4.0 of the External test runner.

There are two types of test runners: in-process and external.

Polylith comes bundled with an in-process test runner that uses Clojure’s testing framework. This is the default test runner.

If you want to run tests with an external test runner, we recommend Sean Corfield’s external test runner.

In-Process Test Runner

As you have learned, the poly tool can run your tests incrementally. It creates an isolated classloader for each project and runs your tests within that classloader. An in-process runner avoids process startup time costs. It runs all tests in the same process.

For in-process test runners, the poly tool determines which bricks and projects are affected, calculates the classpath for each project, creates an isolated classloader, and passes it to the test runner for execution.

The poly tool, by default, uses an in-process test-runner.

External Test Runner

The alternative to the in-process isolated classloader approach is to execute each project’s tests in a separate external Java subprocess. The advantage of this approach is that you have complete control of the test execution, and memory is released after each project when the subprocess terminates. Use an external test runner if you have issues with the in-process test-runner.

The poly tool will still determine the changed bricks and projects and calculate the classpath for each project. However, the external test runner will launch a subprocess to run the tests.

The poly tool will not run the test setup and teardown functions for external test runners. The external test runner must run these functions within the external process it creates.

In-process vs. External Test Runner

WhatIn-Proces Test RunnerExternal Test Runner

Context

Isolated to project

Isolated to project

Approach

Isolated in-process classloaders

Isolated Java subprocesses

Speed

Fast

Process startup overhead per project

Memory usage

Memory is released when the whole test run has finished

Memory is released after each project test run

Setup & Teardown

Handled by the poly tool

Handled by the test runner

Special arguments

class-loader, eval-in-project

process-ns

Additional functions

N/A

external-process-namespace

Use a Custom Test Runner

To use a custom test runner, add it to your poly tool classpath. An ideal place to do this is the :poly alias in your ./deps.edn file:

{:aliases
 {:poly
  {:extra-deps
   {polylith/clj-poly
    {:mvn/version "0.2.18"}

    io.github.seancorfield/polylith-external-test-runner (1)
    {:git/tag "v0.4.0" :git/sha "eb954fe"
     :deps/root "projects/runner"}

    io.github.imrekoszo/polylith-kaocha (2)
    {:git/tag "v0.8.4" :git/sha "f096de8"}
1We’ve added Sean Corfield’s external test runner, as one example
2And Imre Imre Kószó’s Kaocha test runner as another example. Don’t forget to configure the kaocha-wrapper at the project level, see the documentation or this example project.
You can also have your custom test runner within the same Polylith workspace and depend on it via :local/root.

Once you have your test runner on the classpath, add it to our workspace configuration so that the poly tool can use it instead of the default test runner.

You can override the default global test runner under the :test key in your workspace.edn. Here are three examples:

{...
 ; Global test configuration, used as default for every project.
 :test {:create-test-runner [org.corfield.external-test-runner.interface/create]} (1)

 ; Project specific configurations
 :projects {"foo" {:alias "foo"}
            "bar" {:alias "barr"}
            "baz" {:alias "baz"}}}
1Override default global test runner
You can specify multiple test runners. The poly tool will run each test runner for each tested project.

You can override the global test runner for a project. Add a :test key in the project’s configuration in your workspace.edn to select project-specific test runners:

{...
 ; Global test configuration, used as default for every project.
 :test {:create-test-runner [org.corfield.external-test-runner.interface/create]}

 ; Project specific configurations
 :projects {"foo" {:alias "foo"
                   ; Use another test runner only for this project
                   :test  {:create-test-runner [polylith-kaocha.test-runner/create]}}

            "bar" {:alias "bar"
                   ; Use the poly default test runner instead of the global test-runner
                   :test  {:create-test-runner [:default]}}

            "baz" {:alias "bz"
                   ; Use both the poly default and the example test runner for this project
                   :test {:create-test-runner [:default
                                               org.corfield.external-test-runner.interface/create]}}}}

Implement Your Own Custom Test Runner

Polylith supports writing test runners with two protocols: TestRunner and ExternalTestRunner. You can use these protocols to easily plugin custom test runner(s) into the poly tool.

See our relevant API docs:

To implement your custom test runner, create a single-arity constructor function that reifies the TestRunner protocol. Optionally, reify the ExternalTestRunner protocol to make an external test runner.

(ns se.example.example-test-runner)

...

(defn create [{:keys [workspace project test-settings is-verbose color-mode changes]}]
  ...

  (reify
    test-runner-contract/TestRunner
    (test-runner-name [this] ...)

    (test-sources-present? [this] ...)

    (tests-present? [this runner-opts] ...)

    (run-tests [this runner-opts] ...)

    ; Optional, only if you want an external test runner
    test-runner-contract/ExternalTestRunner
    (external-process-namespace [this] ...)))

The poly tool will call your constructor function to get an instance of your test runner for each project test run. The constructor function will receive a map as the single argument:

KeyDescription

:workspace

The workspace map. This map contains :user-input, which you can use to specify additional arguments for runtime configuration.

:project

A map containing details for the project that poly is currently testing.

:test-settings

Test settings for the project that poly is currently testing. The poly tool extracts this information from workspace.edn.

:is-verbose

When true the poly tool is running tests in verbose mode.

:color-mode

The color mode under which the poly tool is currently running.

:changes

A map of changes since the last stable point in time.

Test configuration

If you create your own test runner or use one from the community, it can be useful to add snippets of test configuration when running the test command.

The test snippets can be configured in workspace.edn under :test-configs (see the test-runners example):

{
 ...

 :test {:create-test-runner org.corfield.external-test-runner.interface/create}

 :test-configs {:default-test-runner  {:create-test-runner [:default]}
                :external-test-runner {:create-test-runner [org.corfield.external-test-runner.interface/create]}
                :kaocha-test-runner   {:create-test-runner [polylith-kaocha.test-runner/create]}
                :exclude-dummy        {:org.corfield/external-test-runner {:focus {:exclude [:dummy]}}}
                :exclude-integration  {:org.corfield/external-test-runner {:focus {:exclude [:integration]}}}}}
...
}

If we run the test command, e.g. poly test, then the test configuration specified by the :test key will be passed to the test runner(s). Let’s start a shell and verify this, from the examples/test-runners directory:

poly

Now you can check the content of the :test key:

test-runners$ ws get:settings:test
{:create-test-runner org.corfield.external-test-runner.interface/create}

Yes, it looked the same. Let’s use the default test runner:

test-runners$ ws get:settings:test with:default-test-runner
{:create-test-runner [:default]}

The old org.corfield.external-test-runner.interface/create value was replaced by [:default].

If the old value instead was set to [org.corfield.external-test-runner.interface/create] then the two vectors would instead been merged into [org.corfield.external-test-runner.interface/create :defalt].

Both default-test-runner and kaocha-test-runner store their values in a vector, and if we select both, the result will be merged:

test-runners$ ws get:settings:test with:default-test-runner:kaocha-test-runner
{:create-test-runner [:default polylith-kaocha.test-runner/create]}

Here is another example, where :integration and :dummy are concatenated:

test-runners$ ws get:settings:test with:exclude-integration:exclude-dummy
{:create-test-runner org.corfield.external-test-runner.interface/create,
 :org.corfield/external-test-runner {:focus {:exclude [:integration :dummy]}}}

Now when we understand how test configuration can be added, we can run the test command using the default test runner:

test-runners$ test with:default-test-runner

Test Runners from the Community

The default test runner works fine in most cases and is simple and fast. Sometimes, using the same classloader for all your tests in the workspace doesn’t give enough isolation. In this case, the External Test Runner is a good choice. If you switch to the Kaocha Test Runner, you will get more options for running your tests.

Kaocha Test Runner

A simple Kaocha-based test runner implementation for the poly tool.

Type

in-process

Repository

imrekoszo/polylith-kaocha

Author

@imrekoszo

License

MIT

External Test Runner

An external (subprocess) test runner for Polylith. Avoids classloader, daemon thread, and memory usage issues by running tests in a subprocess with only Clojure itself as a dependency.

Type

external

Repository

seancorfield/polylith-external-test-runner

Author

@seancorfield

License

Apache-2.0

Can you improve this documentation?Edit on GitHub

cljdoc is a website building & hosting documentation for Clojure/Script libraries

× close