Liking cljdoc? Tell your friends :D

Developer Guide

Supported Environments

Rewrite-clj is verified on each push on macOS, Ubuntu and Windows via GitHub Actions.

All scripts are written in Clojure and most invoked via babashka. This gives us a cross platform scripting language that is familiar, fun and consistent. These docs will show babashka scripts invoked explicitly via babashka’s bb; on macOS and linux feel free to leave out bb.

We make use of planck for cljs bootstrap (aka cljs self-hosted) testing. Planck is currently not available for Windows.

We test that rewrite-clj operates as expected when natively compile via GraalVM. Automated testing is setup using GraalVM v21 JDK11.

Prerequisites

Java JDK 1.8 or above
NodeJs v12 or above
Clojure v1.10.1.697 or above for clojure command
- Note that rewrite-clj v1 itself supports Clojure v1.8 and above
Babashka v0.3.7 or above
GraalVM v21.2.0 JDK 11 (if you want to run GraalVM native image tests)

Windows Notes

Git and newlines

The primary development OSes for rewrite-clj are macOS and Linux. Our line endings are LF only.

I’m not sure what Windows developers typically want for line endings while working on source. I expect, but don’t know, that most Windows editors automatically handle LF as line ending. Someone let me know if I am wrong.

Note that I do explicitly set git’s config core.autocrlf to false on our Windows CI unit test environment. Our import vars code generation checks currently rely on line endings remaining unconverted.

Babashka

The Clojure story on Windows is still in the early chapters. Scoop offers an easy way to install tools. @littleli is doing a great job w/maintaining scoop apps for Clojure, Babashka and other tools and this is how I installed Babashka.

Clojure

We all choose our own paths, but for me, using deps.clj instead of Clojure’s PowerShell Module offered me no fuss no muss Clojure on Windows and GitHub Actions on Windows. I decided to install deps.clj not through scoop but through the deps.clj install.ps1 script. This makes it simple to treat deps.exe as if it were the official clojure via a simple rename:

Rename-Item $HOME\deps.clj\deps.exe clojure.exe

GraalVM

You’ll have your own preference, but I find it convenient to install GraalVM on Windows via scoop.

You’ll need to load the appropriate Visual C++ environment variables for GraalVM’s native-image to do its work. I found it oddly cumbersome to load them from PowerShell, so I work from a cmd shell instead. Here’s what works on my Windows dev environment:

call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat"

Setup

After checking out this project from GitHub,

Install JavaScript libraries and tools required by doo and shadow-cljs:
```
sudo npm install karma-cli -g
npm install
```
If you are on macOS or linux, install planck.
Initialize cache for clj-kondo so it can lint against your dependencies
```
bb lint
```

Babashka Tasks

We make use of babashka tasks for development related commands.

To see all available tasks with a short description run:

bb tasks

To run a task, for example, the lint task:

bb lint

Usage help for a task is requested via --help, for example:

bb lint --help

Tasks are described throughout this document.

Code Generation

Rewrite-clj v0 used a version of potemkin import-vars. Potemkin import-vars copies specified vars from a specified namespace to the current namespace at load time. Due to often mysterious issues related to import-vars, a general dislike for import-vars in the Clojure community, and associated maintenance costs, we’ve opted to instead generate code for rewrite-clj v1.

For any source that used potemkin import-vars, we now have a separate template clj (or cljc) file. For example src/rewrite_clj/zip.cljc is generated by template template/rewrite_clj/zip.cljc.

The syntax of import-vars in the template remains familiar. The following old potemkin import-vars syntax:

(import-vars
  [[my.ns1 my-var1 my-var2 my-var3]
   [my.ns2 my-var4 my-var5]])

Is expressed in our templates as:

#_{:import-vars/import
   {:from [[my.ns1 my-var1 my-var2 my-var3]
           [my.ns2 my-var4 my-var5]]}}

Any :added and :deprecated metadata should be defined in the template and not on the reference var. This keeps the metadata on the public API vars only and avoids having the ClojureScript compiler warn about deprecated calls on internal sources within rewrite-clj:

#_{:import-vars/import
   {:from [[my.ns1
            ^{:deprecated "1.2.3"} obsolete-fn
            ^{:added "1.2.4"} new-fn]]}}

We also carry over rewrite-cljc support for :import-vars/import-with-mods, via an optional :opts. See template/rewrite_clj/zip.cljc for example usage.

Importing will generate delegates. An import of (defn foo [a b] (+ a b)) from namespace my.ns1 will generate (defn foo [a b] (my.ns1/foo a b)). No generation of requires is done, your template will have to require my.ns1 in normal Clojure code.

At this time, we don’t handle destructuring in arglists, and will throw unless args are all symbols.

To generate target source from templates run:

bb apply-import-vars gen-code

You are expected to review the generated changes and commit the generated source to version control. We don’t lint templates, but we do lint the generated code.

To perform a read-only check, run:

bb apply-import-vars check

The check command will exit with 0 if no changes are required, otherwise it will exit with 1. Our build script will run the check command and fail the build if there are any pending changes that have not been applied.

Testing During Development

Your personal preference will likely be different, but during maintenance and refactoring, I found running tests continuously for Clojure and ClojureScript helpful.

Clojure

For Clojure, I open a shell terminal window and run:

bb test-clj-watch

This launches kaocha in watch mode.

ClojureScript

For ClojureScript, I open a shell terminal window and run:

bb test-cljs-watch

This launches fighweel main. After initialization, your default web browser will automatically be opened with the figwheel auto-testing page.

Docs

All documentation is written in AsciiDoc. We follow AsciiDoc best practice of one sentence per line.

Images are created and edited with draw.io desktop. We export to .png with a border of 10 and a transparent background. At the time of this writing draw.io does not remember export settings, so you’ll have to enter them in each time.

Testing Doc Code Blocks

We use test-doc-blocks to verify that code blocks in our documentation are in good working order.

bb test-doc

This generates tests for doc code blocks and then runs them under Clojure and ClojureScript.

Testing Before a Push

Before pushing, you likely want to mimic what is run on each push via GitHub Actions.

Unit tests

Unit tests are run via:

bb ci-unit-tests

Native image tests

We also verify that rewrite-clj functions as expected when compiled via Graal’s native-image.

Tests and library natively compiled:
```
bb test-native
```
Library natively compiled and tests interpreted via sci
```
bb test-native-sci
```

Libs test

To try to ensure our changes to rewrite-clj do not inadvertently break existing popular libraries, we run their tests, or a portion thereof, against rewrite-clj.

bb test-libs run

See README for current libs we test against.

Additional libs are welcome.

To see a list of available libs we currently test against:

bb test-libs list

If you are troubleshooting locally, and want to only run specific tests, you can specify which ones you’d like to run. For example:

bb test-libs run cljfmt zprint

Updating the test-libs script to run against current versions of libs is recommended, but care must be taken when updating. We want to make sure we are patching correctly to use rewrite-clj v1 and running a lib’s tests as intended.

To check for outdated libs:

bb test-libs outdated

Notes:

The test-libs task was developed on macOS and is run on CI under Linux only under JDK 11 only. We can expand variations at some later date if there is any value to it.
We test the current HEAD of rewrite-clj v1 against specific versions (latest at the time of this writing) of libs.
We patch lib deps and sometimes code (ex. require for rewrite-cljc becomes rewrite-clj).
As folks migrate to rewrite-clj v1, the need for current patches will lessen.
Updating what versions we test against is currently a manual, but not an overly burdensome, task.

Checking for Outdated Dependencies

To see what new dependencies are available, run:

bb outdated

This task uses:

antq for Clojure.
npm for JavaScript. It only checks against installed ./node_modules, so you may want to run npm install first.

Linting

We use clj-kondo and eastwood to lint rewrite-clj source code.

We fail the build on any lint violations. The CI server runs:

bb lint

and you can too. The lint script will build the clj-kondo cache when it is missing or stale. If you want to force a rebuild of the cache run:

bb lint --rebuild-cache

Integrate clj-kondo into your editor to catch mistakes as they happen.

You can optionally:

bb -lint-kondo to only run clj-kondo linter
bb -lint-eastwood to only run eastwood linter

API diffs

Rewrite-clj v1’s primary goals include remaining compatible with rewrite-clj v0 and rewrite-cljs and avoiding breaking changes.

To generate reports on differences between rewrite-clj v0, rewrite-cljs and rewrite-clj v1 APIs, run:

bb doc-api-diffs

This task currently needs love, see #132.

Run this script manually on an as-needed basis, and certainly before any official release. Generated reports are to be checked in to version control.

Reports are generated to doc/generated/api-diffs/ and include manually written notes from doc/diff-notes/.

These reports are referenced from other docs, so if you rename files, be sure to search for links.

Makes use of diff-apis. Delete .diff-apis/.cache if you need a clean run.

Cljdoc Preview

Before a release, it can be comforting to preview what docs will look like on cljdoc.

Limitations

This task should be considered experimental, I have only tested running on macOS, but am fairly confident it will work on Linux. Not sure about Windows at this time.
You have to push your changes to GitHub to preview them. This allows for a full preview that includes any links (source, images, etc) to GitHub. This works fine from branches and forks - in case you don’t want to affect your main development branch for a preview.

Start Local Services

To start the local cljdoc docker container:

bb cljdoc-preview start

The local cljdoc server allows your ingested docs to be viewed in your web browser.

The start command also automatically checks docker hub for any updates so that our cljdoc preview matches the current production version of cljdoc.

Ingest Docs

To ingest rewrite-clj API and docs into the local cljdoc database:

bb cljdoc-preview ingest

The ingest command automatically publishes rewrite-clj to your local maven repository (cljdoc only works with published jars).

The locally published version will include a -cljdoc-preview suffix. I find this distinction helps to reduce confusion around locally vs remotely installed artifacts.

You’ll have to remember to git commit and git push your changes before ingesting.

Repeat these steps any time you want to preview changes.

Preview Docs

To open a view to the ingested docs in your default web browser:

bb cljdoc-preview view

If you have just run the start command, be a bit patient, the cljdoc server can take a few moments to start up - especially on macOS due to poor file sharing performance.

Stop Local Services

When you are done, you’ll want to stop your docker container:

bb cljdoc-preview stop

This will also delete temporary files created to support your preview session, most notably the local cljdoc database.

Note that NO cleanup is done for any rewrite-clj artifacts published to your local maven repository.

Container Status

If you forget where you are at with your docker containers, run:

bb cljdoc-preview status

Code Coverage

We use cloverage via kaocha to generate code coverage reports via:

bb test-coverage

Our CI service is setup to automatically generate then upload reports to CodeCov.

We have no specific goals for code coverage, but new code is generally expected to have tests.

So why measure coverage? It simply offers us some idea of what code our test suite hits.

Contributors

We honor current and past contributors to rewrite-clj in our README file.

To update contributors, update doc/contributors.edn then run:

bb doc-update-readme

❮User Guide Design❯

Can you improve this documentation? These fine people already did:
lread & Vincent CantinEdit on GitHub

cljdoc is a website building & hosting documentation for Clojure/Script libraries

Keyboard shortcuts Report a problem cljdoc on GitHub

× close