Rich Comment Tests (RCT)

RCT turns rich comment forms into tests.

^:rct/test
(comment
  (+ 1 1) ;=> 2
  (+ 1 1) ;=> 3
  )

(com.mjdowney.rich-comment-tests/run-ns-tests! *ns*)
; Testing com.mjdowney.rich-comment-tests.example
;
; FAIL in () (example.clj:4)
; expected: (= (+ 1 1) 3)
;   actual: (not (= 2 3))
;
; Ran 1 tests containing 2 assertions.
; 1 failures, 0 errors.
;=> {:test 1, :pass 1, :fail 1, :error 0}

Coordinates

CHANGELOG | Uses Break Versioning

io.github.robertluo/rich-comment-tests #:git{:sha "22ea82d321b99f659f18c5b8822859292bd1026cn"}

Introduction

RCT is a version of the excellent hyperfiddle/rcf that uses rewrite-clj to evaluate comment blocks and match the result of each sexpr against ;=> result comments.

It was inspired by the discussion in hyperfiddle/rcf/issues/49. Further discussion / feature requests welcome.

Its goals are to encourage writing rich comment forms in the most natural way possible, using normal (comment ...) forms and Clojure comments, and to integrate nicely with both REPL and clojure.test workflows.

Non-goals include providing advanced unit test features and syntax (the original hyperfiddle/rcf is much better for this!) or completely replacing clojure.test. I see this as a complementary tool to aid in REPL development, help keep examples from comment forms up to date with CI, and encourage writing small tests alongside the function under test.

Usage

1. Write rich comment forms as you normally would, and tag some of them with {:rct/test true}.
2. Run these tests during development by sending run-ns-tests! to the REPL (I have an editor shortcut that reloads the namespace and then does this).
3. Configure your test runner to run all the rct tests in your source tree (see also: Use with Babashka Tasks).

Assertions

RCT supports three kinds of assertions:

• => asserts literal equality
• =>> asserts a matcho pattern (and allows ... to indicate a partial pattern)
• throws=>> asserts an exception expected, the thrown exception (Throwable) will turns into a error record, like {:error/class Exception, :error/message "this is an error", :error/cause #{another...} :error/data {:some 'data}}, an example: (throw (ex-info "ok" {:number 3})) ;throws=>> #:error{:message #".." :data {:number odd?}}

Assertions are either part of the comment or follow it directly.

Extending assertions

RCT treats a symbol directly followed by => or =>> as an assertion operator. To make a new assertion operator, refer to emit-tests/emit-assertion multimethod.

^:rct/test
(comment
  ;; Literal assertions with =>
  (range 3) ;=> (0 1 2)
  (+ 5 5) ;; => 10
  (System/getProperty "java.version.date") ;=> "2022-09-20"


  ;; Pattern matching assertions with =>>
  (range 3) ;=>> '(0 1 ...)
  (+ 5 5) ;=>> int?

  (into {} (System/getProperties))
  ;=>> {"java.version.date" #"\d{4}-\d{2}-\d{2}"}

  (def response {:status 200 :body "ok"})
  response
  ;=>> {:status #(< % 300)
  ;     :body   not-empty}

  ;; Or with spec
  (require '[clojure.spec.alpha :as s])
  (into {} (System/getProperties)) ;=>> (s/map-of string? string?)

  ;; Or using a blank ;=> line to match against the next form
  response
  ;=>
  {:status 200
   :body "ok"}
  )

Use with clojure.test

RCT is designed to hook in nicely with clojure.test reporting / assertion counting, and to be easy to run from an idiomatic Clojure CI workflow.

Option 1: Use a single deftest to run all rich comment tests in the source tree

This works well for the following scenario:

• you have a test/ directory with source files that use clojure.test + deftest,
• plus a src directory with rich comment tests sprinkled throughout,
• ... and you want all your tests to run at once.

(ns some-test-ns
  (:require [clojure.test :refer :all]
    [com.mjdowney.rich-comment-tests.test-runner :as test-runner]))

(deftest rct-tests
  (test-runner/run-tests-in-file-tree! :dirs #{"src"}))

Now when you run clojure.test, rich comment tests are also included.

This is what happens when you run this project with:

clj -X:test

Option 2: Only run rich comment tests

For a project that only uses rich comment tests, you can add an alias to deps.edn:

{:aliases
 {:test {:exec-fn com.mjdowney.rich-comment-tests.test-runner/run-tests-in-file-tree!
         :exec-args {:dirs #{"src"}}}}}

Use with Babashka tasks

Sample bb.edn file to run RCTs via bb test:

{:paths ["src"]
 :deps  {}
 :tasks {test
         {:docs "Run unit tests."
          :extra-deps {io.github.matthewdowney/rich-comment-tests {...}}
          :requires ([com.mjdowney.rich-comment-tests.test-runner :as rct])
          :task (rct/run-tests-in-file-tree! {:dirs #{"src"}})}}}

See also: Running tests from the Babashka book.

Changes

This fork

• Introduces throws=>> operator to catch expected Throwable(Exceptions).
• Exchange the order of => form to match the (is (= .. ..)) concept.
• Fix a bug that namespaced keywords can not be correctly resolved.
• Fix the standalone test runner, it will have an exit code of 1 if some tests do not succeed.

v1.0.2 (2023-02-09)

• Update for Babashka test/report-counters is a ref instead of an atom for bb >= 1.1.171 #18

v1.0.1 (2023-02-07)

• Fix: Babashka run-tests-in-file-tree! throws: No implementation of method: :getName of protocol: #'sci.impl.types/HasName #17
• Make it easier to find the line from which test exceptions are thrown

v1.0.0 — no breaking changes, API is now stable 🎉 (2023-01-21)

• Fix: ;=> nil assertions ignored #16
• Add stdout capture helper #15

v0.0.4 (2023-01-04)

• Add Babashka support #14
• Allow non-commented expectation strings following blank ;=> line #13
• Make context string parsing smarter #8
• Add CI for PRs / pushes to main #10

v0.0.3 (2022-12-11)

• (@lilactown) Automatically quote result when used with => #5
• (@lilactown) Allow whitespace between semicolon and arrow (;; =>) #3
• Support matcho assertions with ;=>> #2
• Allow expectation string ellipses like ;=>> {:a :b ...} #1

v0.0.2 (2022-12-07)

• Add integration with clojure.test reporting + way to run RCT alongside it
• Improve resolution of source file path from Clojure namespace
• Add support for multi-line result comments

v0.0.1 (2022-12-06)

• Initial working version.