Liking cljdoc? Tell your friends :D

Developer Guide

Supported Environments

Rewrite-clj is verified on each push on macOS, Ubuntu and Windows via GitHub Actions.

All scripts are written in Clojure and most invoked via babashka. This gives us a cross platform scripting language that is familiar, fun and consistent. These docs will show babashka scripts invoked explicitly via babashka’s bb; on macOS and linux feel free to leave out bb.

We make use of planck for cljs bootstrap (aka cljs self-hosted) testing. Planck is currently not available for Windows.

We test that rewrite-clj operates as expected when natively compile via GraalVM. Automated testing is setup using GraalVM v21 for both JDK8 and JDK11. On Windows we only test against JDK11 as tool setup for JDK8 on Windows seemed overly arduous.

Prerequisites

  • Java JDK 1.8 or above

  • NodeJs v12 or above

  • Clojure v1.10.1.697 or above for clojure command

    • Note that rewrite-clj v1 itself supports Clojure v1.9 and above

  • Babashka v0.3.1 or above

  • GraalVM v21.0.0 (if you want to run GraalVM native image tests)

Windows Notes

Git and newlines

The primary development OSes for rewrite-clj are macOS and Linux. Our line endings are LF only.

I’m not sure what Windows developers typically want for line endings while working on source. I expect, but don’t know, that most Windows editors automatically handle LF as line ending. Someone let me know if I am wrong.

Note that I do explicitly set git’s config core.autocrlf to false on our Windows CI unit test environment. Our import vars code generation checks currently rely on line endings remaining unconverted.

Babashka

The Clojure story on Windows is still in the early chapters. Scoop offers an easy way to install tools. @littleli is doing a great job w/maintaining scoop apps for Clojure, Babashka and other tools and this is how I installed Babashka.

Clojure

We all choose our own paths, but for me, using deps.clj instead of Clojure’s PowerShell Module offered me no fuss no muss Clojure on Windows and GitHub Actions on Windows. I decided to install deps.clj not through scoop but through the deps.clj install.ps1 script. This makes it simple to treat deps.exe as if it were the official clojure via a simple rename:

Rename-Item $HOME\deps.clj\deps.exe clojure.exe

GraalVM

You’ll have your own preference, but I find it convenient to install GraalVM on Windows via scoop. Both graalvm11 and graalvm8 are available.

You’ll need to load the appropriate Visual C++ environment variables for GraalVM’s native-image to do its work. I found it oddly cumbersome to load them from PowerShell, so I work from a cmd shell instead. Here’s what works on my Windows dev environment:

call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat"

And finally, I never did figure out how to get the Windows prerequisites setup for the JDK8 version of GraalVM, so I only test on the JDK11 version.

Setup

After checking out this project from GitHub,

  1. Install JavaScript libraries and tools required by doo and shadow-cljs:

    sudo npm install karma-cli -g
npm install

  2. If you are on macOS or linux, install planck.

  3. Initialize cache for clj-kondo so it can lint against your dependencies

    bb ./script/lint.clj

Code Generation

Rewrite-clj v0 used a version of potemkin import-vars. Potemkin import-vars copies specified vars from a specified namespace to the current namespace at load time. Due to often mysterious issues related to import-vars, a general dislike for import-vars in the Clojure community, and associated maintenance costs, we’ve opted to instead generate code for rewrite-clj v1.

For any source that used potemkin import-vars, we now have a separate template clj (or cljc) file. For example src/rewrite_clj/zip.cljc is generated by template template/rewrite_clj/zip.cljc.

The syntax of import-vars in the template remains familiar. The following old potemkin import-vars syntax:

(import-vars
  [[my.ns1 my-var1 my-var2 my-var3]
   [my.ns2 my-var4 my-var5]])

Is expressed in our templates as:

#_{:import-vars/import
   {:from [[my.ns1 my-var1 my-var2 my-var3]
           [my.ns2 my-var4 my-var5]]}}

We also carry over rewrite-cljc support for :import-vars/import-with-mods, via an optional :opts. See template/rewrite_clj/zip.cljc for example usage.

Importing will generate delegates. An import of (defn foo [a b] (+ a b)) from namespace my.ns1 will generate (defn foo [a b] (my.ns1/foo a b)). No generation of requires is done, your template will have to require my.ns1 in normal Clojure code.

At this time, we don’t handle destructuring in arglists, and will throw unless args are all symbols.

To generate target source from templates run:

bb script/apply_import_vars.clj gen-code

You are expected to review the generated changes and commit the generated source to version control. We don’t lint templates, but we do lint the generated code.

To perform a read-only check, run:

bb script/apply_import_vars.clj check

The check command will exit with 0 if no changes are required, otherwise it will exit with 1. Our build script will run the check command and fail the build if there are any pending changes that have not been applied.

Testing During Development

Your personal preference will likely be different, but during maintenance and refactoring, I found running tests continuously for Clojure and ClojureScript helpful.

Clojure

For Clojure, I open a shell terminal window and run:

bb ./script/clj_watch.clj

This launches kaocha in watch mode.

ClojureScript

For ClojureScript, I open a shell terminal window and run:

bb ./script/cljs_watch.clj

This launches fighweel main. After initialization, your default web browser will automatically be opened with the figwheel auto-testing page.

Docs

All documentation is written in AsciiDoc. We follow AsciiDoc best practice of one sentence per line.

Images are created and edited with draw.io desktop. We export to .png with a border of 10 and a transparent background. At the time of this writing draw.io does not remember export settings, so you’ll have to enter them in each time.

Testing Doc Code Blocks

We use test-doc-blocks to verify that code blocks in our documentation are in good working order.

bb ./script/doc_tests.clj

This generates tests for doc code blocks and then runs them under Clojure and ClojureScript.

Testing Before a Push

Before pushing, you likely want to mimic what is run on each push via GitHub Actions.

Unit tests

Unit tests are run via:

bb ./script/ci_tests.clj

Native image tests

We also verify that rewrite-clj functions as expected when compiled via Graal’s native-image.

  1. Tests and library natively compiled:

    bb ./script/pure_native_test.clj

  2. Library natively compiled and tests interpreted via sci

    bb ./script/sci_native_test.clj

Libs test

To try to ensure our changes to rewrite-clj do not inadvertently break existing popular libraries, we run their tests, or a portion thereof, against rewrite-clj.

bb ./script/libs_tests.clj run

See README for current libs we test against.

Additional libs are welcome.

If you are troubleshooting locally, and want to only run specific tests, you can specify which ones you’d like to run. For example:

bb ./script/libs_tests.clj run cljfmt zprint

Running current versions of libs is recommended, but care must be taken when updating. We want to make sure we are patching correctly to use rewrite-clj v1 and running a lib’s tests as intended.

To check for outdated libs:

bb ./script/libs_test.clj outdated

Notes:

  • libs_tests.clj was developed on macOS and is run on CI under Linux only under JDK 11 only. We can expand variations at some later date if there is any value to it.

  • We test the current HEAD of rewrite-clj v1 against specific versions (latest at the time of this writing) of libs.

  • We patch lib deps and sometimes code (ex. require for rewrite-cljc becomes rewrite-clj).

  • As folks migrate to rewrite-clj v1, the need for current patches will lessen.

  • Updating what versions we test against is currently a manual, but not an overly burdensome, task.

Checking for Outdated Dependencies

To see what new dependencies are available, run:

bb ./script/outdated.clj

We use antq which also checks pom.xml. If you see an outdated dependency reported for pom.xml after updating deps.edn, run the following:

clojure -Spom

This script also checks for outdated Node.js dependencies. Note that checks are only done against installed ./node_modules, so you may want to run npm install first.

Linting

We use clj-kondo for linting rewrite-clj source code.

We fail the build on any lint violations. The CI server runs:

bb ./script/lint.clj

and you can too.

Integrate clj-kondo into your editor to catch mistakes as they happen.

API diffs

Rewrite-clj v1’s primary goals include remaining compatible with rewrite-clj v0 and rewrite-cljs and avoiding breaking changes.

To generate reports on differences between rewrite-clj v0, rewrite-cljs and rewrite-clj v1 APIs, run:

bb ./script/gen_api_diffs.clj

Run this script manually on an as-needed basis, and certainly before any official release. Generated reports are to be checked in to version control.

Reports are generated to doc/generated/api-diffs/ and include manually written notes from doc/diff-notes/.

These reports are referenced from other docs, so if you rename files, be sure to search for links.

Makes use of diff-apis. Delete .diff-apis/.cache if you need a clean run.

Cljdoc Preview

Before a release, it can be comforting to preview what docs will look like on cljdoc.

Limitations

  • This script should be considered experimental, I have only tested running on macOS, but am fairly confident it will work on Linux. Not sure about Windows at this time.

  • You have to push your changes to GitHub to preview them. This allows for a full preview that includes any links (source, images, etc) to GitHub. This works fine from branches and forks - in case you don’t want to affect your main development branch for a preview.

Start Local Services

To start the local cljdoc docker container:

bb ./script/cljdoc_preview.clj start

The local cljdoc server allows your ingested docs to be viewed in your web browser.

The start command also automatically checks docker hub for any updates so that our cljdoc preview matches the current production version of cljdoc.

Ingest Docs

To ingest rewrite-clj API and docs into the local cljdoc database:

bb ./script/cljdoc_preview.clj ingest

The ingest command automatically publishes rewrite-clj to your local maven repository (cljdoc only works with published jars). You’ll have to remember to git commit and git push your changes before ingesting.

Repeat these steps any time you want to preview changes.

Preview Docs

To open a view to the ingested docs in your default web browser:

bb ./script/cljdoc_preview.clj view

If you have just run the start command, be a bit patient, the cljdoc server can take a few moments to start up - especially on macOS due to poor file sharing performance.

Stop Local Services

When you are done, you’ll want to stop your docker container:

bb ./script/cljdoc_preview.clj stop

This will also delete temporary files created to support your preview session, most notably the local cljdoc database.

Note that NO cleanup is done for any rewrite-clj artifacts published to your local maven repository.

Container Status

If you forget where you are at with your docker containers, run:

bb ./script/cljdoc_preview.clj status

Code Coverage

We use cloverage via kaocha to generate code coverage reports. Our CI service is setup to automatically generate then upload reports to CodeCov.

We have no specific goals for code coverage, but new code is generally expected to have tests.

So why measure coverage? It simply offers us some idea of what code our test suite hits.

Contributors

We honor current and past contributors to rewrite-clj in our README file.

To update contributors, update doc/contributors.edn then run:

clojure -M:update-readme
❮User GuideDesign❯

Can you improve this documentation? These fine people already did:
lread & Vincent CantinEdit on GitHub

cljdoc is a website building & hosting documentation for Clojure/Script libraries

Keyboard shortcutsReport a problemcljdoc on GitHub
× close