Readme — borkdude/babashka 0.1.0

Liking cljdoc? Tell your friends :D

A Clojure babushka for the grey areas of Bash.

Life's too short to remember how to write Bash code. I feel liberated.
— @laheadle on Clojurians Slack

Introduction

The main idea behind babashka is to leverage Clojure in places where you would be using bash otherwise.

As one user described it:

I’m quite at home in Bash most of the time, but there’s a substantial grey area of things that are too complicated to be simple in bash, but too simple to be worth writing a clj/s script for. Babashka really seems to hit the sweet spot for those cases.

Goals

Low latency Clojure scripting alternative to JVM Clojure.
Easy installation: grab the self-contained binary and run. No JVM needed.
Familiarity and portability:
- Scripts should be compatible with JVM Clojure as much as possible
- Scripts should be platform-independent as much as possible. Babashka offers support for linux, macOS and Windows.
Allow interop with commonly used classes like java.io.File and System
Multi-threading support (pmap, future, core.async)
Connectivity: talk UDP, TCP, HTTP (and optionally: JDBC)
Support for various data formats: JSON, XML, YAML, CSV, bencode
Batteries included (tools.cli, cheshire, ...)
Library support via popular tools like the clojure CLI

Non-goals

Performance¹
Provide a mixed Clojure/Bash DSL (see portability).
Replace existing shells. Babashka is a tool you can use inside existing shells like bash and it is designed to play well with them. It does not aim to replace them.

^{1^{Babashka uses sci for
interpreting Clojure. Sci implements a suffiently large subset of
Clojure. Interpreting code is in general not as performant as executing compiled
code. If your script takes more than a few seconds to run, Clojure on the JVM
may be a better fit, since the performance of Clojure on the JVM outweighs its
startup time penalty. Read more about the differences with Clojure
here.}}

Talk

To get an overview of babashka, you can watch this talk (slides):

Quickstart

$ curl -s https://raw.githubusercontent.com/borkdude/babashka/master/install -o install-babashka
$ chmod +x install-babashka && ./install-babashka
$ ls | bb -i '(filter #(-> % io/file .isDirectory) *input*)'
("doc" "resources" "sci" "script" "src" "target" "test")
bb took 4ms.

Examples

Read the output from a shell command as a lazy seq of strings:

$ ls | bb -i '(take 2 *input*)'
("CHANGES.md" "Dockerfile")

Read EDN from stdin and write the result to stdout:

$ bb '(vec (dedupe *input*))' <<< '[1 1 1 1 2]'
[1 2]

Read more about input and output flags here.

Execute a script. E.g. print the current time in California using the java.time API:

File pst.clj:

#!/usr/bin/env bb

(def now (java.time.ZonedDateTime/now))
(def LA-timezone (java.time.ZoneId/of "America/Los_Angeles"))
(def LA-time (.withZoneSameInstant now LA-timezone))
(def pattern (java.time.format.DateTimeFormatter/ofPattern "HH:mm"))
(println (.format LA-time pattern))

$ pst.clj
05:17

More examples can be found here.

Status

Functionality regarding clojure.core and java.lang can be considered stable and is unlikely to change. Changes may happen in other parts of babashka, although we will try our best to prevent them. Always check the release notes or CHANGES.md before upgrading.

Installation

Brew

Linux and macOS binaries are provided via brew.

Install:

brew install borkdude/brew/babashka

Upgrade:

brew upgrade babashka

Arch (Linux)

babashka is available in the Arch User Repository. It can be installed using your favorite AUR helper such as yay, yaourt, apacman and pacaur. Here is an example using yay:

yay -S babashka-bin

Windows

On Windows you can install using scoop and the scoop-clojure bucket.

Installer script

Install via the installer script:

$ bash <(curl -s https://raw.githubusercontent.com/borkdude/babashka/master/install)

By default this will install into /usr/local/bin. To change this, provide the directory name:

$ bash <(curl -s https://raw.githubusercontent.com/borkdude/babashka/master/install) /tmp

Download

You may also download a binary from Github. For linux there is a static binary available which can be used on Alpine.

Docker

Check out the image on Docker hub.

Usage

Babashka v0.0.90

Options must appear in the order of groups mentioned below.

Help:

  --help, -h or -?    Print this help text.
  --version           Print the current version of babashka.
  --describe          Print an EDN map with information about this version of babashka.

In- and output flags:

  -i                  Bind *input* to a lazy seq of lines from stdin.
  -I                  Bind *input* to a lazy seq of EDN values from stdin.
  -o                  Write lines to stdout.
  -O                  Write EDN values to stdout.
  --stream            Stream over lines or EDN values from stdin. Combined with -i or -I *input* becomes a single value per iteration.

Uberscript:

  --uberscript <file> Collect preloads, -e, -f and -m and all required namespaces from the classpath into a single executable file.

Evaluation:

  -e, --eval <expr>   Evaluate an expression.
  -f, --file <path>   Evaluate a file.
  -cp, --classpath    Classpath to use.
  -m, --main <ns>     Call the -main function from namespace with args.
  --verbose           Print entire stacktrace in case of exception.

REPL:

  --repl              Start REPL. Use rlwrap for history.
  --socket-repl       Start socket REPL. Specify port (e.g. 1666) or host and port separated by colon (e.g. 127.0.0.1:1666).
  --nrepl-server      Start nREPL server. Specify port (e.g. 1667) or host and port separated by colon (e.g. 127.0.0.1:1667).

If neither -e, -f, or --socket-repl are specified, then the first argument that is not parsed as a option is treated as a file if it exists, or as an expression otherwise. Everything after that is bound to *command-line-args*. Use -- to separate script command lin args from bb command line args.

The clojure.core functions are accessible without a namespace alias.

The following namespaces are required by default and available through the pre-defined aliases in the user namespace. You may use require + :as and/or :refer on these namespaces. If not all vars are available, they are enumerated explicitly.

clojure.string aliased as str
clojure.set aliased as set
clojure.edn aliased as edn:
- read-string
clojure.java.shell aliased as shell
clojure.java.io aliased as io:
- as-relative-path, as-url, copy, delete-file, file, input-stream, make-parents, output-stream, reader, resource, writer
clojure.main: repl
clojure.core.async aliased as async.
clojure.stacktrace
clojure.test
clojure.pprint: pprint (currently backed by fipp's fipp.edn/pprint)
clojure.tools.cli aliased as tools.cli
clojure.data.csv aliased as csv
clojure.data.xml aliased as xml
cheshire.core aliased as json
cognitect.transit aliased as transit
clj-yaml.core alias as yaml
bencode.core aliased as bencode: read-bencode, write-bencode
next.jdbc aliased as jdbc (available under feature flag)

A selection of java classes are available, see babashka/impl/classes.clj.

Babashka supports import: (import clojure.lang.ExceptionInfo).

Babashka supports a subset of the ns form where you may use :require and :import:

(ns foo
  (:require [clojure.string :as str])
  (:import clojure.lang.ExceptionInfo))

For the unsupported parts of the ns form, you may use reader conditionals to maintain compatibility with JVM Clojure.

Running a script

Scripts may be executed from a file using -f or --file:

bb -f download_html.clj

Using bb with a shebang also works:

#!/usr/bin/env bb

(defn get-url [url]
  (println "Fetching url:" url)
  (let [{:keys [:exit :err :out]} (shell/sh "curl" "-sS" url)]
    (if (zero? exit) out
      (do (println "ERROR:" err)
          (System/exit 1)))))

(defn write-html [file html]
  (println "Writing file:" file)
  (spit file html))

(let [[url file] *command-line-args*]
  (when (or (empty? url) (empty? file))
    (println "Usage: <url> <file>")
    (System/exit 1))
  (write-html file (get-url url)))

$ ./download_html.clj
Usage: <url> <file>

$ ./download_html.clj https://www.clojure.org /tmp/clojure.org.html
Fetching url: https://www.clojure.org
Writing file: /tmp/clojure.org.html

If /usr/bin/env doesn't work for you, you can use the following workaround:

$ cat script.clj
#!/bin/sh

#_(
   "exec" "bb" "$0" hello "$@"
   )

(prn *command-line-args*)

./script.clj 1 2 3
("hello" "1" "2" "3")

Input and output flags

In one-liners the *input* value may come in handy. It contains the input read from stdin as EDN by default. If you want to read in text, use the -i flag, which binds *input* to a lazy seq of lines of text. If you want to read multiple EDN values, use the -I flag. The -o option prints the result as lines of text. The -O option prints the result as lines of EDN values.

Note: *input* is only available in the user namespace, on other namespaces use *in*.

The following table illustrates the combination of options for commands of the form

echo "{{Input}}" | bb {{Input flags}} {{Output flags}} "*input*"

Input	Input flags	Output flag	`input`	Output
`{:a 1}` `{:a 2}`			`{:a 1}`	`{:a 1}`
hello bye	`-i`		`("hello" "bye")`	`("hello" "bye")`
hello bye	`-i`	`-o`	`("hello" "bye")`	hello bye
`{:a 1}` `{:a 2}`	`-I`		`({:a 1} {:a 2})`	`({:a 1} {:a 2})`
`{:a 1}` `{:a 2}`	`-I`	`-O`	`({:a 1} {:a 2})`	`{:a 1}` `{:a 2}`

When combined with the --stream option, the expression is executed for each value in the input:

$ echo '{:a 1} {:a 2}' | bb --stream '*input*'
{:a 1}
{:a 2}

Current file path

The var *file* contains the full path of the file that is currently being executed:

$ cat example.clj
(prn *file*)

$ bb example.clj
"/Users/borkdude/example.clj"

Command-line arguments

Command-line arguments can be retrieved using *command-line-args*. If you want to parse command line arguments, you may use the built-in clojure.tools.cli namespace or use the nubank/docopt library.

Additional namespaces

babashka.classpath

Contains the function add-classpath which can be used to add to the classpath dynamically:

(require '[babashka.classpath :refer [add-classpath]]
         '[clojure.java.shell :refer [sh]])
(def medley-dep '{:deps {medley {:git/url "https://github.com/borkdude/medley"
                                 :sha "91adfb5da33f8d23f75f0894da1defe567a625c0"}}})
(def cp (:out (sh "clojure" "-Spath" "-Sdeps" (str medley-dep))))
(add-classpath cp)
(require '[medley.core :as m])
(m/index-by :id [{:id 1} {:id 2}]) ;;=> {1 {:id 1}, 2 {:id 2}}

babashka.wait

Contains the functions: wait-for-port and wait-for-path.

Usage of wait-for-port:

(wait/wait-for-port "localhost" 8080)
(wait/wait-for-port "localhost" 8080 {:timeout 1000 :pause 1000})

Waits for TCP connection to be available on host and port. Options map supports :timeout and :pause. If :timeout is provided and reached, :default's value (if any) is returned. The :pause option determines the time waited between retries.

Usage of wait-for-path:

(wait/wait-for-path "/tmp/wait-path-test")
(wait/wait-for-path "/tmp/wait-path-test" {:timeout 1000 :pause 1000})

Waits for file path to be available. Options map supports :default, :timeout and :pause. If :timeout is provided and reached, :default's value (if any) is returned. The :pause option determines the time waited between retries.

The namespace babashka.wait is aliased as wait in the user namespace.

babashka.signal

Contains the function signal/pipe-signal-received?. Usage:

(signal/pipe-signal-received?)

Returns true if PIPE signal was received. Example:

$ bb '((fn [x] (println x) (when (not (signal/pipe-signal-received?)) (recur (inc x)))) 0)' | head -n2
1
2

The namespace babashka.signal is aliased as signal in the user namespace.

babashka.curl

The namespace babashka.curl is a tiny wrapper around curl. It's aliased as curl in the user namespace. See babashka.curl.

Style

A note on style. Babashka recommends the following:

Explicit requires

Use explicit requires with namespace aliases in scripts, unless you're writing one-liners.

Do this:

$ ls | bb -i '(-> *input* first (str/includes? "m"))'
true

But not this:

script.clj:

(-> *input* first (str/includes? "m"))

Rather do this:

script.clj:

(ns script
  (:require [clojure.java.io :as io]
            [clojure.string :as str]))
  (-> (io/reader *in*) line-seq first (str/includes? "m"))

Some reasons for this:

Linters like clj-kondo work better with code that uses namespace forms, explicit requires, and known Clojure constructs
Editor tooling works better with namespace forms (sorting requires, etc).
Writing compatible code gives you the option to run the same script with clojure

Running a REPL

Babashka offers a REPL, a socket REPL and an nREPL server. Look here for more information on how to use and integrate them with your editor.

Preloads

The environment variable BABASHKA_PRELOADS allows to define code that will be available in all subsequent usages of babashka.

BABASHKA_PRELOADS='(defn foo [x] (+ x 2))'
BABASHKA_PRELOADS=$BABASHKA_PRELOADS' (defn bar [x] (* x 2))'
export BABASHKA_PRELOADS

Note that you can concatenate multiple expressions. Now you can use these functions in babashka:

$ bb '(-> (foo *input*) bar)' <<< 1
6

You can also preload an entire file using load-file:

export BABASHKA_PRELOADS='(load-file "my_awesome_prelude.clj")'

Note that *input* is not available in preloads.

Classpath

Babashka accepts a --classpath option that will be used to search for namespaces and load them:

$ cat src/my/namespace.clj
(ns my.namespace)
(defn -main [& _args]
  (println "Hello from my namespace!"))

$ bb --classpath src --main my.namespace
Hello from my namespace!

So if you have a larger script with a classic Clojure project layout like

$ tree -L 3
├── deps.edn
├── README
├── src
│   └── project_namespace
│       ├── main.clj
│       └── utilities.clj
└── test
    └── project_namespace
        ├── test_main.clj
        └── test_utilities.clj

Then you can tell Babashka to include both the src and test folders in the classpath and start a socket REPL by running:

$ bb --classpath src:test --socket-repl 1666

Note that you can use the clojure tool to produce classpaths and download dependencies:

$ cat deps.edn
{:deps
 {my_gist_script
  {:git/url "https://gist.github.com/borkdude/263b150607f3ce03630e114611a4ef42"
   :sha "cfc761d06dfb30bb77166b45d439fe8fe54a31b8"}}
 :aliases {:my-script {:main-opts ["-m" "my-gist-script"]}}}

$ CLASSPATH=$(clojure -Spath)
$ bb --classpath "$CLASSPATH" --main my-gist-script
Hello from gist script!

If there is no --classpath argument, the BABASHKA_CLASSPATH environment variable will be used:

$ export BABASHKA_CLASSPATH=$(clojure -Spath)
$ export BABASHKA_PRELOADS="(require '[my-gist-script])"
$ bb "(my-gist-script/-main)"
Hello from gist script!

When invoking bb with a main function, the expression (System/getProperty "babashka.main") will return the name of the main function.

Also see the babashka.classpath namespace which allows dynamically adding to the classpath.

See deps.clj for a babashka script that replaces the clojure bash script.

Data readers

Data readers can be enabled by setting *data-readers* to a hashmap of symbols to functions or vars:

$ bb "(set! *data-readers* {'t/tag inc}) #t/tag 1"
2

To preserve good startup time, babashka does not scan the classpath for data_readers.clj files.

Uberscript

The --uberscript option collects the expressions in BABASHKA_PRELOADS, the command line expression or file, the main entrypoint and all required namespaces from the classpath into a single file. This can be convenient for debugging and deployment.

Given the deps.edn from above:

$ deps.clj -A:my-script -Scommand "bb -cp {{classpath}} {{main-opts}} --uberscript my-script.clj"

$ cat my-script.clj
(ns my-gist-script)
(defn -main [& args]
  (println "Hello from gist script!"))
(ns user (:require [my-gist-script]))
(apply my-gist-script/-main *command-line-args*)

$ bb my-script.clj
Hello from gist script!

Parsing command line arguments

Babashka ships with clojure.tools.cli:

(require '[clojure.tools.cli :refer [parse-opts]])

(def cli-options
  ;; An option with a required argument
  [["-p" "--port PORT" "Port number"
    :default 80
    :parse-fn #(Integer/parseInt %)
    :validate [#(< 0 % 0x10000) "Must be a number between 0 and 65536"]]
   ["-h" "--help"]])

(:options (parse-opts *command-line-args* cli-options))

$ bb script.clj
{:port 80}
$ bb script.clj -h
{:port 80, :help true}

Reader conditionals

Babashka supports reader conditionals by taking either the :bb or :clj branch, whichever comes first. NOTE: the :clj branch behavior was added in version 0.0.71, before that version the :clj branch was ignored.

$ bb "#?(:bb :hello :clj :bye)"
:hello

$ bb "#?(:clj :bye :bb :hello)"
:bye

$ bb "[1 2 #?@(:bb [] :clj [1])]"
[1 2]

Running tests

Babashka bundles clojure.test. To make CI scripts fail you can use a simple runner like this:

#!/usr/bin/env bash
bb -cp "src:test:resources" \
   -e "(require '[clojure.test :as t] '[borkdude.deps-test])
       (let [{:keys [:fail :error]} (t/run-tests 'borkdude.deps-test)]
         (System/exit (+ fail error)))"

Spawning and killing a process

Use the java.lang.ProcessBuilder class.

Example:

user=> (def ws (-> (ProcessBuilder. ["python" "-m" "SimpleHTTPServer" "1777"]) (.start)))
#'user/ws
user=> (wait/wait-for-port "localhost" 1777)
{:host "localhost", :port 1777, :took 2}
user=> (.destroy ws)
nil

Also see this example.

Async

In addition to future, pmap, promise and friends, you may use the clojure.core.async namespace for asynchronous scripting. The following example shows how to get first available value from two different processes:

bb '
(defn async-command [& args]
  (async/thread (apply shell/sh "bash" "-c" args)))

(-> (async/alts!! [(async-command "sleep 2 && echo process 1")
                   (async-command "sleep 1 && echo process 2")])
    first :out str/trim println)'
process 2

Note: the go macro is available for compatibility with JVM programs, but the implementation maps to clojure.core.async/thread and the single exclamation mark operations (<!, >!, etc.) map to the double exclamation mark operations (<!!, >!!, etc.). It will not "park" threads, like on the JVM.

HTTP

For making HTTP requests you can use:

babashka.curl. This library is included with babashka and aliased as curl in the user namespace.
slurp for simple GET requests
clj-http-lite as a library.
clojure.java.shell or java.lang.ProcessBuilder for shelling out to your favorite command line http client

HTTP over Unix sockets

This can be useful for talking to Docker:

(require '[clojure.java.shell :refer [sh]])
(require '[cheshire.core :as json])
(-> (sh "curl" "--silent"
        "--no-buffer" "--unix-socket"
        "/var/run/docker.sock"
        "http://localhost/images/json")
    :out
    (json/parse-string true)
    first
    :RepoTags) ;;=> ["borkdude/babashka:latest"]

Shutdown hook

Adding a shutdown hook allows you to execute some code before the script exits.

$ bb -e '(-> (Runtime/getRuntime) (.addShutdownHook (Thread. #(println "bye"))))'
bye

This also works when the script is interrupted with ctrl-c.

JDBC

Babashka supports the next.jdbc library along with drivers for PostgresQL and HSQLDB. These features are not part of the standard bb distribution. See doc/build.md for details on how to build babashka with these features. See this test for an example how to use this.

Interacting with psql, mysql and the sqlite CLIs can be achieved by shelling out. See the examples directory.

Bencode

Babashka comes with the nrepl/bencode library which allows you to read and write bencode messages to a socket. A simple example which evaluates a Clojure expression on an nREPL server started with lein repl:

(ns nrepl-client
  (:require [bencode.core :as b]))

(defn nrepl-eval [port expr]
  (let [s (java.net.Socket. "localhost" port)
        out (.getOutputStream s)
        in (java.io.PushbackInputStream. (.getInputStream s))
        _ (b/write-bencode out {"op" "eval" "code" expr})
        bytes (get (b/read-bencode in) "value")]
    (String. bytes)))

(nrepl-eval 52054 "(+ 1 2 3)") ;;=> "6"

Differences with Clojure

Babashka is implemented using the Small Clojure Interpreter. This means that a snippet or script is not compiled to JVM bytecode, but executed form by form by a runtime which implements a sufficiently large subset of Clojure. Babashka is compiled to a native binary using GraalVM. It comes with a selection of built-in namespaces and functions from Clojure and other useful libraries. The data types (numbers, strings, persistent collections) are the same. Multi-threading is supported (pmap, future).

Differences with Clojure:

A pre-selected set of Java classes are supported. You cannot add Java classes at runtime.
Interpretation comes with overhead. Therefore tight loops are likely slower than in Clojure on the JVM. In general interpretation yields slower programs than compiled programs.
No defprotocol, defrecord and unboxed math.

Libraries, pods and projects

A list of projects (scripts, libraries, pods and tools) known to work with babashka.

Pods

Pods are programs that can be used as a Clojure library by babashka. Documentation is available in the library repo.

Package babashka script as a AWS Lambda

AWS Lambda runtime doesn't support signals, therefore babashka has to disable handling of the SIGPIPE. This can be done by setting BABASHKA_DISABLE_PIPE_SIGNAL_HANDLER to true.

Articles, podcasts and videos

Babashka Pods presentation by Michiel Borkent at the Dutch Clojure Meetup.
AWS Logs using Babashka a blog published by toyokumo.
The REPL podcast Michiel Borkent talks about clj-kondo, Jet, Babashka, and GraalVM with Daniel Compton.
Implementing an nREPL server for babashka: impromptu presentation by Michiel Borkent at the online Dutch Clojure Meetup
ClojureScript podcast with Jacek Schae interviewing Michiel Borkent
Babashka talk at ClojureD (slides) by Michiel Borkent
Babashka: a quick example by Malcolm Sparks
Clojure Start Time in 2019 by Stuart Sierra
Advent of Random Hacks by Arne Brasseur
Clojure in the Shell by Arne Brasseur
Clojure Tool by Eric Normand

Building babashka

Developing Babashka

Including new libraries or classes

Before new libraries or classes go into the standardly distributed babashka binary, these evaluation criteria are considered:

The library or class is useful for general purpose scripting.
Adding the library or class would make babashka more compatible with Clojure libraries relevant to scripting.
The library cannot be interpreted by with babashka using --classpath.
The functionality can't be met by shelling out to another CLI or can't be written as a small layer over an existing CLI (like babashka.curl) instead.

If not all of the criteria are met, but adding a feature is still useful to a particular company or niche, adding it behind a feature flag is still a possibility. This is currently the case for next.jdbc and the PostgresQL and HSQLDB database drivers. Companies interested in these features can compile an instance of babashka for their internal use. Companies are also free to make forks of babashka and include their own internal libraries. If their customized babashka is interesting to share with the world, they are free to distribute it using a different binary name (like bb-sql, bb-docker, bb-yourcompany, etc.). See the feature flag documentation and the implementation of the existing feature flags (example commit).

Related projects

Examples

A collection of example scripts.

Thanks

adgoji for financial support
CircleCI for CI and additional support
Nikita Prokopov for the logo
contributors and other users posting issues with bug reports and ideas

Contributors

Code Contributors

This project exists thanks to all the people who contribute. [Contribute].

Financial Contributors

Become a financial contributor and help us sustain our community. [Contribute]

Individuals

Organizations

Support this project with your organization. Your logo will show up here with a link to your website. [Contribute]

License

Distributed under the EPL License. See LICENSE.

This project contains code from:

Clojure, which is licensed under the same EPL License.

Can you improve this documentation? These fine people already did:
Michiel Borkent, Arne Brasseur, Sameer Kolhar, sogaiu, Rovanion Luckey, Peter Nagy, Dainius Jocas, Nikita Prokopov, Martin Klepsch, Will, Victor Bjelkholm, Peter Strömberg, Gabriel Horner, Nate Sutton, jess & David HarriganEdit on GitHub

cljdoc is a website building & hosting documentation for Clojure/Script libraries

Keyboard shortcuts Report a problem cljdoc on GitHub

× close