Liking cljdoc? Tell your friends :D

lread.test-doc-blocks

Test Clojars cljdoc

Generate Clojure tests from AsciiDoc and CommonMark doc code blocks.

Example code in docs, while likely not malicious, could be illustrating, for example, how to wipe a drive.
So, do be deliberate and careful about what docs you run through test-doc-blocks.

Status

Initial concept. Currently being used to test doc blocks in rewrite-clj.

  • Alpha, this might not be a good general idea, but might work well for some projects. It is working very well for rewrite-clj.

  • Feedback welcome.

Rationale

I wanted to make sure the code examples I provided in rewrite-clj v1 documentation do not mislead and function as expected for those who dare to try them.

Test-doc-blocks might (or might not, see limitations, and interesting alternatives) be of interest for your Clojure project too.

If your example code blocks are REPL style:

user=> (* 6 7)
42

Or editor style:

(* 6 7)
;; => 42

And don’t rely on any special unspecified setup, then test-doc-blocks might work for your project too.

Usage

Add an alias to your deps.edn:

    :gen-doc-tests {:extra-deps {com.github.lread/test-doc-blocks {:mvn/version "1.0.124-alpha"}}
                    :exec-fn lread.test-doc-blocks/gen-tests}

Then the most basic usage is:

clojure -X:gen-doc-tests

This generates Clojure tests for code blocks in your README.md file to the target/test-doc-blocks/test directory. Any existing tests under target/test-doc-blocks will be replaced.

The generated tests have no dependency on test-doc-blocks and can be run with your preferred test runner.

Test-doc-blocks looks for assertions in your doc code blocks that are in editor style and REPL style format. For example…​

user=> (/ 714 17)
42

…​generates the test assertion (is (= 42 (/ 714 17))). For more detailed examples and inline options, see:

The test-doc-blocks deps.edn has some aliases that might serve as useful examples:

  • :isolated/cljs-test-runner - runs generated tests under ClojureScript using cljs-test-runner
    Invoke for this project via: clj -M:isolated/cljs-test-runner

  • :isolated/kaocha - runs generated tests under Clojure using kaocha
    Invoke for this project via: clj -M:isolated/kaoacha generated. Note also kaocha tests.edn config.

  • :isolated/clj-test-runner - runs generated tests under Clojure using Cognitect test-runner
    Invoke for this project via: clj -M:isolated/clj-test-runner

Test generation requires Clojure v1.9 or later.
Generated tests have been verified to run against Clojure v1.8 or later - except under kaocha which itself requires Clojure v1.9 or later.

Command Line Options

:docs

The default file tests are generated for is README.md.

If you want to specify a different vector of files you can do so via :docs:

clojure -X:gen-doc-tests :docs '["README.adoc" "doc/example.adoc" "doc/example.md" "doc/example.cljc"]'

:docs understands glob syntax.

:target-root

The default directory to generate tests is ./target.

You can override this via :target-root:

clojure -X:gen-doc-tests :target-root '"./someplace/else"'

Note that test-doc-blocks will delete and recreate test-docs-block/test dirs under the target root. Keep that the location mind when figuring out where to point your test runner.

:platform

The platform governs what Clojure file types will be generated for tests.

Specify:

  • :clj for Clojure, generates .clj files

  • :cljs for ClojureScript, generates .cljs files

  • :cljc for mixed, generates .cljc files

The default is :cljc. Platform can be overridden for code blocks via inline options.
No platform assumptions are currently made when generating test from doc blocks found in Clojure source files. Specify what makes sense for your tests.

Windows users, trying to correctly escape -X args on the command line can lead to frustration. Consider instead specifying options under :exec-args in your :gen-doc-tests alias in deps.edn.

Limitations

Some current limitations that we might entertain addressing:

  • If your code block depends on some external setup, we’ve no way to express that.

  • Test-doc-blocks will automatically handle inline (require …​) and (import …​) appearing in code blocks, but not in any complex expressions of these forms.

  • Parsing adoc and md files is on the naive side but should handle most common cases. If we’ve overlooked a common syntax, let us know.

Some limitations we have no current plans to address:

  • Code blocks using ns or in-ns will not work with test-doc-blocks.

  • For REPL style code blocks we only look for user⇒ prompts and no other ns prompts.

  • It is possible to embed HTML into your docs. If your code or headings are expressed in embedded HTML within your doc, test-doc-blocks won’t find them.

Interesting Alternatives

Other options and related projects that I am currently aware of:

  • kaocha - Kaocha supports running cucumber tests. It uses this support in tests for its own documentation. A .feature document describes the feature and includes given, when, then scenarios that are both run and shown in documentation. Gritty details can be hidden in step definitions.

  • readme - Generates tests for code blocks found in .md files and then runs them.

  • testdoc - Tests code blocks in docstrings and external docs.

  • alc.x-as-tests - Runs code in (comment …​) blocks as tests.

Development

Prerequisites

In addition to the Clojure CLI, you will need to install babashka to run scripts.

The example ClojureScript test runner makes use of Node.js

Tests

Unit

clojure -M:kaocha unit

Integration

Run integration test via:

clojure -M:kaocha integration

This will generate tests for README and example docs and then diff against a previously manually verified test run. The previously verified test run is stored under test-resources/expected.

On failure careful manual inspection is recommended. When you are happy with current behaviour of generation of tests:

bb script/gen_local_tests.clj regen-expected

And commit/push the expected results.

Generated

Generate tests for local docs via:

bb script/gen_local_tests.clj

Run generated tests under Clojure via:

clojure -M:isolated/kaocha generated

And under ClojureScript via:

clojure -M:isolated/cljs-test-runner

Continuous Integration

To run what CI runs:

bb script/ci_tests.clj

Linting

We use clj-kondo to lint project source and fail the build when linting fails.

To run linting as the CI server does:

bb script/lint.clj

Versioning

rewrite-cljc versioning scheme is: major.minor.patch-test-qualifier

  • major increments when the API has been broken - something, as a rule, we’d like to avoid.

  • minor increments to convey significant new features have been added.

  • patch indicates bug fixes - it is the number of commits in the repo.

  • test-qualifier is absent for stable releases. Can be alpha, beta, rc1, etc.

The project maintainer manually manages major minor and test-qualifier in version.edn.

Releasing

A release is triggered manually via a GitHub Action "Release" workflow. A release can currently only be cut at main branch HEAD.

GitHub Actions prompts for the branch when running the Release workflow. I’ve not found a way to disable this, just leave it set to main.

At this time, the release workflow does not run tests. The assumption is that you’ve waited for the last CI test run to complete and are happy with the results.

Our GitHub Actions "Release" workflow:

  1. Calculates version (commit count will compensate for upcoming release commit) and applies to:

    1. the deps.edn usage example in this doc

    2. the "unreleased" headings in change log (new "unreleased heading also prepended")

    3. pom.xml version and scm→tag

    4. a newly built thin jar

  2. Deploys jar to clojars

  3. Commits updated docs and release tag back to repo

  4. Informs cljdoc of the new release

  5. Creates a release tag at HEAD

After a release don’t forget to pull the changes made by the release workflow.

As a developer you can run the following locally:

  • bb script/release.clj validate - to validate your change log is setup appropriately for release

  • bb script/release.clj prep - to complete all steps up to, but not including, deploy, note that this will affect changes to README.adoc, CHANGELOG.adoc and pom.xml, you’ll NOT want to commit these changes.

People

Contributors

  • @seancorfield - idea is based on Sean’s readme project.

  • Helpful feedback and ideas from:

    • @borkdude

    • @sogaiu

    • @dominicin

    • @pez

    • @seancorfield

    • @uochan

Current Maintainers

  • @lread

License

Copyright © 2021 Lee Read, all rights reserved.

Distributed under the EPL License, same as Clojure. See LICENSE.

Concept based on @seancorfield’s readme which is distributed under EPL v1.0 or later.

Can you improve this documentation? These fine people already did:
lread & GitHub ActionsEdit on GitHub

cljdoc is a website building & hosting documentation for Clojure/Script libraries

Keyboard shortcutsReport a problemcljdoc on GitHub
× close