Design Notes — com.github.lread/test-doc-blocks 1.1.20

Design Notes

2021-Aug

Support :refer-clojure?

While adding test-doc-blocks to honeysql, I noticed its code blocks use (refer-clojure :exclude '[…]).

Background

Test-doc-blocks does move up supported namespace function calls up into the generated test ns declarations. It currently does this for (require …) and (import …) calls only. For example:

(require '[clojure.edn :as edn])
(import '[MyThing one])

Will be included in the target test ns as:

(ns my-generate-test-ns
  (:require [clojure.edn :as edn]
            <... other test based requires ...>)
  (:import [MyThing one])

This is done to support ClojureScript which does not support these functions inline at runtime.

Originally I was thinking we did not need to bother with (refer-clojure …) calls, I thought the result would be some compiler warnings that we could ignore when running generated tests.

But… I think we might need to address this as cljc tests were failing for ClojureScript without the explicit clojure core exclusions. Somehow there was an issue with a HoneySql helper referred for that was not overidding clojure core’s for without the explicit clojure core exclusion.

Strategy

When a doc contains code blocks with multiple require and import calls, test-doc-blocks will do its best to amalgamate them when moving the to the ns declaration.

Amalgamation does not make sense for multiple refer-clojure calls. It would confuse more than help.

A doc author will want to either:

include one refer-clojure in a code block at the top of their doc that applies to all code blocks in the doc
Or include multiple code blocks with different refer-clojure calls, but in this case use test-doc-block/ns inline option to selectively target refer-clojure calls to specific code blocks.

Test-doc-blocks will helpfully fail test generation if it finds more than one refer-clojure call per platform in a generated test ns.

2021-Jan

Thoughts and Ramblings

After playing with my December experiment, I decided to take an alternate approach. I no longer use macros to parse and run tests at test runtime. I instead convert REPL assertions to deftest assertions a test generation time.

The disadvantage is that the generated tests look less like the original doc code blocks.

Advantages are:

I can freely bring in useful dependencies to help with the test generation. So far I am experimenting with clojure.tools.reader and rewrite-cljc.
The generated tests look like normal clojure tests and can be easily reasoned about without studying a custom macros.

Other changes:

No longer trying to automatically handle violaters of pr (like rewrite-clj). The work-around is to spit output to stdout and use =stdout⇒ assertion.
I was prefixing generated test namespaces with test-doc-blocks.gen. The idea was to provide isolation from other code. Since the test-ns is under the control of the author (with the default being based on the doc filename), this perhaps presumptuous.
I had some primitive support for bringing up inline requires into the ns declaration (to support ClojureScript). It is still primitive, but now I also support bringing up inline imports into the ns declaration. And do some recognizing of these elements when they, or parts of their bodies are wrapped in reader conditionals.
Although I liked having one deftest with multiple testings, because it made for good looking test reports, I decided to only have one testing per deftest. This will allow for tagging of individual code blocks with metadata, which can then in turn be used by test runners.
=stdout⇒ and =stderr⇒ continuation is now single ; rather than ;;. This allows us to distinguish output from assertion tokens so we can have things like:
```
(println "oh my\n;; =>")
;; =stdout=>
; oh my
; ;; =>
```

Still to ponder:

Sean’s readme project doesn’t wrap code blocks without assertions into tests. Test-doc-blocks currently does. My thinking was that it would be nice to get some context from the test runner on assert-less errant code blocks. But… I might have been wrong. Sean’s solution allows for setup subsequent tests and mine does not.
UPDATE: Actually I think we are probably good here as long as tests are run in order.

Feedback from community

I mentioned to @seancorfield that I was working on test-doc-blocks and he kindly enlisted as a the second alpha tester (after me). He gave it a whirl on honeysql and when he found a few problems I offered to give the honeysql develop branch a whirl and report back. I/we found:

Default cmd line args were borked. Fixed.
Bad :doc cmd line arg error message was too vague. Fixed.
Require/import refs were not being correctly migrated to ns when more than single. Fixed.
A difference in test-doc-blocks and the readme project is that test-doc-blocks always quotes expectations. We thought that maybe evaling the expectation might useful if only rarely. It turns out we don’t need this for honeysql after all so I’m going to hold off for now.
I decided to try to run honeysql readme through cljs test runner too (readme project only did clj). The README (require …) had to be updated for cljs. And this made me think that an inline reader-conditional option might be nice. Sometimes you want to show something for Clojure or ClojureScript users but don’t want to show the reader conditional in your code block.

I think this might have been from last days of December but I’m only recording it now:

@sogaiu Kindly mentioned that my original idea for stdout markers did not allow for the output to contain the marker. Oops! He also prefers 80 chars width so was suggesting using up as little horizontal screen real estate as possible.
@pez Mentioned that stdout and stderr will be printed as they occur for a REPL flow. (I’m ignoring this observation for now).

2020-Dec

Initial Thoughts

Heavily based on, and code used from, @seancorfield’s readme.

Sean’s solution is an all-in-one. Tests are generated run, then deleted. Since I am supporting cljs in addition to clj, I think I’m going to separate generate from test. This is less convenient, but also lets you use your test runner of choice.

Sean’s readme project supports REPL and edit style code blocks. I’ll add support for more editor style code blocks and add some syntax for verifying stdout. See examples docs for examples.

Some test blocks might have conflicting (require …) with other test blocks in the same document. I’ll allow for test blocks to be isolated to their own test namespace.

To support ClojureScript, I’ll need to merge the (require …) into the generated test (ns .. (:require …)). Reasoning: ClojureScript inline (require …) are REPL only - as far as I understand.

Sean makes use of a macro to convert extracted code blocks to tests at runtime. I find this very interesting.

I considered, that instead of using a macro like Sean does, to rewrite code blocks to tests using rewrite-cljc at generation time. I decided against this because I like minimizing deps for a test tool. One could argue that bringing in extra deps for generation only and not for testing is ok, but I want to leave the door open for those who might want to merge these two steps. So I’ll stick with Sean’s macro technique for now.

I’ll make use of (testing …) macro for descriptive tests that show file, maybe last section header and line number. Opting to group all testing blocks under a single deftest has the advantage of cleaner output when running tests in a verbose mode, but also has the disadvantage of finding compilation errors originating from code blocks. Clojure will report the error at the deftest line rather than the testing line.

Writing macros that work for Clojure and ClojureScript is a bit of a challenge, but mostly due to my learning curve, rather than any technical limitations.

Running code blocks can be a potentially dangerous thing. For a contrived example, an author might provide a code snippit to illustrate how to delete all files on your system. So, although it might be convenient, I’ll try to force caution by:

only processing files explicitly listed.
by default, only generate tests for code blocks that have been tagged to be processed. This can be overriden with an invocation option such as {:test :all}, maybe the default will be :tagged-only. Or maybe this is too much? Yeah, the user has listed the file, that’s good enough. We’ll add a big warning in the README.

Continueing with safety, we’ll be generating potentially more than one file. To avoid accidental deletion of other files, we’ll

fail on soft links in deletion set
always create the same root dir under a specified target dir and delete recursively from there.

I’ll start with support for asciidoctor, because this what I write my documents in, but will likely quickly move to also supporting CommonMark.

Generated tests will take advantage of (testing …) to provide a rich descriptive string for the code block under test. The string might contain:

filename
last section header before code block
line number

Options will be optionally conveyed via comments in docs. We’ll take inspiration for clj-kondo for syntax.

In Asciidoctor this will look like: // {:test-doc-blocks/my-option x}
In CommonMark: <-- {:test-doc-blocks/my-option x} -->

I could bring in full featured parsers for Asciidoctor and CommonMark but this would mean extra unwanted deps. I’ll instead implement simple parsing that should be good enough in most cases.

Feedback from community

I asked some folks about any existing conventions for what I sketched as =stdout⇒ and got back some great responses:

@sogaiu

one thing to consider might be whether you want to be able to distinguish between return values and output. it wasn’t clear to me from the snippets above how that might occur.
another consideration might be if you want to verify stdout vs stderr (though arranging for this might be tricky depending on how things are done underneath?). with your sketch, perhaps you’d use "=stderr⇒" as a prefix?
i was also interested in verying output but didn’t set anything down. in my case i’ve tried to make the notation concise with the idea that this stuff might be entered manually. thus i’ve steered clear of longer things if possible.
on a related note, i took a quick peek at test-doc-blocks, but didn’t find anything about supporting expected return values that might be more nicely formatted across lines. in alc.x-as-tests' case, i used #_ to prefix multiline constructs: https://github.com/sogaiu/alc.x-as-tests/blob/master/doc/comment-block-tests.md (search for "discard")

@uochan

FWIW, I also have a similar library to test codes in docstring mainly. But it is also usable for external documents like markdown or asciidoc. https://github.com/liquidz/testdoc

@dominicm

Vim fireplace prefixes lines with ; for stdout

@pez

We have a similar problem in the Calva output window. So far only prefixing both stdout and stderr with ; . But I have been wanting to start using maybe ;o and ;e .

We then dug into clearly representing eval vs stdout and stderr a bit.

Other

One person kindly warned me privately this project might be a bit of a rabbit hole. I see their point, but my main customer is rewrite-cljc, and it already found issues there, so I’ll carry on.