Liking cljdoc? Tell your friends :D

CircleCI Clojars Project project chat

A Clojure babushka for the grey areas of Bash.

Life's too short to remember how to write Bash code. I feel liberated.

@laheadle on Clojurians Slack

Quickstart

$ bash <(curl -s https://raw.githubusercontent.com/borkdude/babashka/master/install)
$ ls | bb --time -i '(filter #(-> % io/file .isDirectory) *input*)'
("doc" "resources" "sci" "script" "src" "target" "test")
bb took 4ms.

Rationale

The sweet spot for babashka is executing Clojure snippets or scripts in the same space where you would use Bash.

As one user described it:

I’m quite at home in Bash most of the time, but there’s a substantial grey area of things that are too complicated to be simple in bash, but too simple to be worth writing a clj/s script for. Babashka really seems to hit the sweet spot for those cases.

Goals:

  • Fast startup / low latency. This is achieved by compiling to native using GraalVM.
  • Familiarity and portability. Keep migration barriers between bash and Clojure as low as possible by:
    • Gradually introducing Clojure expressions to existing bash scripts
    • Scripts written in babashka should also be able to run on the JVM without major changes.
  • Multi-threading support similar to Clojure on the JVM
  • Batteries included (clojure.tools.cli, core.async, ...)

Non-goals:

  • Performance
  • Provide a mixed Clojure/bash DSL (see portability).
  • Replace existing shells. Babashka is a tool you can use inside existing shells like bash and it is designed to play well with them. It does not aim to replace them.

Babashka uses sci for interpreting Clojure. Sci implements a subset of Clojure and is not as performant as compiled code. If your script is taking more than a few seconds, Clojure on the JVM may be a better fit.

Read more about the differences with Clojure here.

Status

Experimental. Breaking changes are expected to happen at this phase. Keep an eye on CHANGES.md for a list of breaking changes.

Examples

$ ls | bb -i '*input*'
["LICENSE" "README.md" "bb" "doc" "pom.xml" "project.clj" "resources" "script" "src" "target" "test"]

$ ls | bb -i '(count *input*)'
12

$ bb '(vec (dedupe *input*))' <<< '[1 1 1 1 2]'
[1 2]

$ bb '(filterv :foo *input*)' <<< '[{:foo 1} {:bar 2}]'
[{:foo 1}]

$ bb '(#(+ %1 %2 %3) 1 2 *input*)' <<< 3
6

$ ls | bb -i '(filterv #(re-find #"README" %) *input*)'
["README.md"]

$ bb '(run! #(shell/sh "touch" (str "/tmp/test/" %)) (range 100))'
$ ls /tmp/test | bb -i '*input*'
["0" "1" "10" "11" "12" "13" "14" "15" "16" "17" "18" "19" "2" "20" "21" ...]

$ bb -O '(repeat "dude")' | bb --stream '(str *input* "rino")' | bb -I '(take 3 *input*)'
("duderino" "duderino" "duderino")

More examples can be found in the gallery.

Installation

Brew

Linux and macOS binaries are provided via brew.

Install:

brew install borkdude/brew/babashka

Upgrade:

brew upgrade babashka

Arch (Linux)

babashka is available in the Arch User Repository. It can be installed using your favorite AUR helper such as yay, yaourt, apacman and pacaur. Here is an example using yay:

yay -S babashka-bin

Windows

On Windows you can install using scoop and the scoop-clojure bucket.

Installer script

Install via the installer script:

$ bash <(curl -s https://raw.githubusercontent.com/borkdude/babashka/master/install)

By default this will install into /usr/local/bin. To change this, provide the directory name:

$ bash <(curl -s https://raw.githubusercontent.com/borkdude/babashka/master/install) /tmp

Download

You may also download a binary from Github.

Usage

Usage: bb [ -i | -I ] [ -o | -O ] [ --stream ] [--verbose]
          [ ( --classpath | -cp ) <cp> ] [ --uberscript <file> ]
          [ ( --main | -m ) <main-namespace> | -e <expression> | -f <file> |
            --repl | --socket-repl [<host>:]<port> ]
          [ arg* ]

Options:

  --help, -h or -?    Print this help text.
  --version           Print the current version of babashka.

  -i                  Bind *input* to a lazy seq of lines from stdin.
  -I                  Bind *input* to a lazy seq of EDN values from stdin.
  -o                  Write lines to stdout.
  -O                  Write EDN values to stdout.
  --verbose           Print entire stacktrace in case of exception.
  --stream            Stream over lines or EDN values from stdin. Combined with -i or -I *input* becomes a single value per iteration.
  --uberscript <file> Collect preloads, -e, -f and -m and all required namespaces from the classpath into a single executable file.

  -e, --eval <expr>   Evaluate an expression.
  -f, --file <path>   Evaluate a file.
  -cp, --classpath    Classpath to use.
  -m, --main <ns>     Call the -main function from namespace with args.
  --repl              Start REPL. Use rlwrap for history.
  --socket-repl       Start socket REPL. Specify port (e.g. 1666) or host and port separated by colon (e.g. 127.0.0.1:1666).
  --time              Print execution time before exiting.
  --                  Stop parsing args and pass everything after -- to *command-line-args*

If neither -e, -f, or --socket-repl are specified, then the first argument that is not parsed as a option is treated as a file if it exists, or as an expression otherwise.
Everything after that is bound to *command-line-args*.

The clojure.core functions are accessible without a namespace alias.

The following namespaces are required by default and available through the pre-defined aliases in the user namespace. You may use require + :as and/or :refer on these namespaces. If not all vars are available, they are enumerated explicitly.

  • clojure.string aliased as str
  • clojure.set aliased as set
  • clojure.edn aliased as edn:
    • read-string
  • clojure.java.shell aliased as shell
  • clojure.java.io aliased as io:
    • as-relative-path, as-url, copy, delete-file, file, input-stream, make-parents, output-stream, reader, resource, writer
  • clojure.main: repl
  • clojure.core.async aliased as async. The alt and go macros are not available but alts!! does work as it is a function.
  • clojure.stacktrace
  • clojure.test
  • clojure.tools.cli aliased as tools.cli
  • clojure.data.csv aliased as csv
  • cheshire.core aliased as json

A selection of java classes are available, see babashka/impl/classes.clj.

Babashka supports import: (import clojure.lang.ExceptionInfo).

Babashka supports a subset of the ns form where you may use :require and :import:

(ns foo
  (:require [clojure.string :as str])
  (:import clojure.lang.ExceptionInfo))

For the unsupported parts of the ns form, you may use reader conditionals to maintain compatibility with JVM Clojure.

Input and output flags

In one-liners the *input* value may come in handy. It contains the input read from stdin as EDN by default. If you want to read in text, use the -i flag, which binds *input* to a lazy seq of lines of text. If you want to read multiple EDN values, use the -I flag. The -o option prints the result as lines of text. The -O option prints the result as lines of EDN values.

The following table illustrates the combination of options for commands of the form

echo "{{Input}}" | bb {{Input flags}} {{Output flags}} "*input*"
InputInput flagsOutput flag*input*Output
{:a 1}
{:a 2}
{:a 1}{:a 1}
hello
bye
-i ("hello" "bye")("hello" "bye")
hello
bye
-i-o("hello" "bye")hello
bye
{:a 1}
{:a 2}
-I ({:a 1} {:a 2})({:a 1} {:a 2})
{:a 1}
{:a 2}
-I-O({:a 1} {:a 2}){:a 1}
{:a 2}

When combined with the --stream option, the expression is executed for each value in the input:

$ echo '{:a 1} {:a 2}' | bb --stream '*input*'
{:a 1}
{:a 2}

Current file path

The var *file* contains the full path of the file that is currently being executed:

$ cat example.clj
(prn *file*)

$ bb example.clj
"/Users/borkdude/example.clj"

Command-line arguments

Command-line arguments can be retrieved using *command-line-args*.

Additional namespaces

babashka.classpath

Contains the function add-classpath which can be used to add to the classpath dynamically:

(require '[babashka.classpath :refer [add-classpath]]
         '[clojure.java.shell :refer [sh]])
(def medley-dep '{:deps {medley {:git/url "https://github.com/borkdude/medley"
                                 :sha "91adfb5da33f8d23f75f0894da1defe567a625c0"}}})
(def cp (:out (sh "clojure" "-Spath" "-Sdeps" (str medley-dep))))
(add-classpath cp)
(require '[medley.core :as m])
(m/index-by :id [{:id 1} {:id 2}]) ;;=> {1 {:id 1}, 2 {:id 2}}

babashka.wait

Contains the functions: wait-for-port and wait-for-path.

Usage of wait-for-port:

(wait/wait-for-port "localhost" 8080)
(wait/wait-for-port "localhost" 8080 {:timeout 1000 :pause 1000})

Waits for TCP connection to be available on host and port. Options map supports :timeout and :pause. If :timeout is provided and reached, :default's value (if any) is returned. The :pause option determines the time waited between retries.

Usage of wait-for-path:

(wait/wait-for-path "/tmp/wait-path-test")
(wait/wait-for-path "/tmp/wait-path-test" {:timeout 1000 :pause 1000})

Waits for file path to be available. Options map supports :default, :timeout and :pause. If :timeout is provided and reached, :default's value (if any) is returned. The :pause option determines the time waited between retries.

The namespace babashka.wait is aliased as wait in the user namespace.

babashka.signal

Contains the function signal/pipe-signal-received?. Usage:

(signal/pipe-signal-received?)

Returns true if PIPE signal was received. Example:

$ bb '((fn [x] (println x) (when (not (signal/pipe-signal-received?)) (recur (inc x)))) 0)' | head -n2
1
2

The namespace babashka.signal is aliased as signal in the user namespace.

Running a file

Scripts may be executed from a file using -f or --file:

bb -f download_html.clj

Files can also be loaded inline using load-file:

bb '(load-file "script.clj")'

Using bb with a shebang also works:

#!/usr/bin/env bb

(defn get-url [url]
  (println "Fetching url:" url)
  (let [{:keys [:exit :err :out]} (shell/sh "curl" "-sS" url)]
    (if (zero? exit) out
      (do (println "ERROR:" err)
          (System/exit 1)))))

(defn write-html [file html]
  (println "Writing file:" file)
  (spit file html))

(let [[url file] *command-line-args*]
  (when (or (empty? url) (empty? file))
    (println "Usage: <url> <file>")
    (System/exit 1))
  (write-html file (get-url url)))

(System/exit 0)
$ ./download_html.clj
Usage: <url> <file>

$ ./download_html.clj https://www.clojure.org /tmp/clojure.org.html
Fetching url: https://www.clojure.org
Writing file: /tmp/clojure.org.html

If /usr/bin/env doesn't work for you, you can use the following workaround:

$ cat script.clj
#!/bin/sh

#_(
   "exec" "bb" "$0" hello "$@"
   )

(prn *command-line-args*)

./script.clj 1 2 3
("hello" "1" "2" "3")

Preloads

The environment variable BABASHKA_PRELOADS allows to define code that will be available in all subsequent usages of babashka.

BABASHKA_PRELOADS='(defn foo [x] (+ x 2))'
BABASHKA_PRELOADS=$BABASHKA_PRELOADS' (defn bar [x] (* x 2))'
export BABASHKA_PRELOADS

Note that you can concatenate multiple expressions. Now you can use these functions in babashka:

$ bb '(-> (foo *input*) bar)' <<< 1
6

You can also preload an entire file using load-file:

export BABASHKA_PRELOADS='(load-file "my_awesome_prelude.clj")'

Note that *input* is not available in preloads.

Classpath

Babashka accepts a --classpath option that will be used to search for namespaces and load them:

$ cat src/my/namespace.clj
(ns my.namespace)
(defn -main [& _args]
  (println "Hello from my namespace!"))

$ bb --classpath src --main my.namespace
Hello from my namespace!

Note that you can use the clojure tool to produce classpaths and download dependencies:

$ cat deps.edn
{:deps
 {my_gist_script
  {:git/url "https://gist.github.com/borkdude/263b150607f3ce03630e114611a4ef42"
   :sha "cfc761d06dfb30bb77166b45d439fe8fe54a31b8"}}
 :aliases {:my-script {:main-opts ["-m" "my-gist-script"]}}}

$ CLASSPATH=$(clojure -Spath)
$ bb --classpath "$CLASSPATH" --main my-gist-script
Hello from gist script!

If there is no --classpath argument, the BABASHKA_CLASSPATH environment variable will be used:

$ export BABASHKA_CLASSPATH=$(clojure -Spath)
$ export BABASHKA_PRELOADS="(require '[my-gist-script])"
$ bb "(my-gist-script/-main)"
Hello from gist script!

Deps.clj

The deps.clj script can be used to work with deps.edn-based projects:

$ deps.clj -A:my-script -Scommand "bb -cp {{classpath}} {{main-opts}}"
Hello from gist script!

Create these aliases for brevity:

$ alias bbk='deps.clj -Scommand "bb -cp {{classpath}} {{main-opts}}"'
$ alias babashka='rlwrap deps.clj -Scommand "bb -cp {{classpath}} {{main-opts}}"'
$ bbk -A:my-script
Hello from gist script!
$ babashka
Babashka v0.0.58 REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

user=> (require '[my-gist-script :as mgs])
nil
user=> (mgs/-main)
Hello from gist script!
nil

Uberscript

The --uberscript option collects the expressions in BABASHKA_PRELOADS, the command line expression or file, the main entrypoint and all required namespaces from the classpath into a single file. This can be convenient for debugging and deployment.

Given the deps.edn from above:

$ deps.clj -A:my-script -Scommand "bb -cp {{classpath}} {{main-opts}} --uberscript my-script.clj"

$ cat my-script.clj
(ns my-gist-script)
(defn -main [& args]
  (println "Hello from gist script!"))
(ns user (:require [my-gist-script]))
(apply my-gist-script/-main *command-line-args*)

$ bb my-script.clj
Hello from gist script!

Parsing command line arguments

Babashka ships with clojure.tools.cli:

(require '[clojure.tools.cli :refer [parse-opts]])

(def cli-options
  ;; An option with a required argument
  [["-p" "--port PORT" "Port number"
    :default 80
    :parse-fn #(Integer/parseInt %)
    :validate [#(< 0 % 0x10000) "Must be a number between 0 and 65536"]]
   ["-h" "--help"]])

(:options (parse-opts *command-line-args* cli-options))
$ bb script.clj
{:port 80}
$ bb script.clj -h
{:port 80, :help true}

Reader conditionals

Babashka supports reader conditionals using the :bb feature:

$ cat example.clj
#?(:clj (in-ns 'foo) :bb (println "babashka doesn't support in-ns yet!"))

$ ./bb example.clj
babashka doesn't support in-ns yet!

Running tests

Babashka bundles clojure.test. To make CI scripts fail you can use a simple runner like this:

#!/usr/bin/env bash
bb -cp "src:test:resources" \
   -e "(require '[clojure.test :as t] '[borkdude.deps-test])
       (let [{:keys [:fail :error]} (t/run-tests 'borkdude.deps-test)]
         (System/exit (+ fail error)))"

REPL

Babashka supports both a REPL and socket REPL. To start the REPL, type:

$ bb --repl

To get history with up and down arrows, use rlwrap:

$ rlwrap bb --repl

To start the socket REPL you can do this:

$ bb --socket-repl 1666
Babashka socket REPL started at localhost:1666

Now you can connect with your favorite socket REPL client:

$ rlwrap nc 127.0.0.1 1666
Babashka v0.0.14 REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

bb=> (+ 1 2 3)
6
bb=> :repl/quit
$

A socket REPL client for Emacs is inf-clojure.

Spawning and killing a process

Use the java.lang.ProcessBuilder class.

Example:

user=> (def ws (-> (ProcessBuilder. ["python" "-m" "SimpleHTTPServer" "1777"]) (.start)))
#'user/ws
user=> (wait/wait-for-port "localhost" 1777)
{:host "localhost", :port 1777, :took 2}
user=> (.destroy ws)
nil

Also see this example.

Async

Apart from future and pmap for creating threads, you may use the async namespace, which maps to clojure.core.async, for asynchronous scripting. The following example shows how to get first available value from two different processes:

bb '
(defn async-command [& args]
  (async/thread (apply shell/sh "bash" "-c" args)))

(-> (async/alts!! [(async-command "sleep 2 && echo process 1")
                   (async-command "sleep 1 && echo process 2")])
    first :out str/trim println)'
process 2

Differences with Clojure

Babashka is implemented using the Small Clojure Interpreter. This means that a snippet or script is not compiled to JVM bytecode, but executed form by form by a runtime which implements a subset of Clojure. Babashka is compiled to a native binary using GraalVM. It comes with a selection of built-in namespaces and functions from Clojure and other useful libraries. The data types (numbers, strings, persistent collections) are the same. Multi-threading is supported (pmap, future).

Differences with Clojure:

  • A subset of Java classes are supported.

  • Only the clojure.core, clojure.edn, clojue.java.io, clojure.java.shell, clojure.set, clojure.stacktrace, clojure.string, clojure.template, clojure.test and clojure.walk namespaces are available from Clojure.

  • Interpretation comes with overhead. Therefore tight loops are likely slower than in Clojure on the JVM.

  • No support for unboxed types.

External resources

Tools and libraries

The following libraries are known to work with Babashka:

deps.clj

A port of the clojure bash script to Clojure / babashka.

spartan.spec

An babashka-compatible implementation of clojure.spec.alpha.

missing.test.assertions

This library checks if no assertions have been made in a test:

$ export BABASHKA_CLASSPATH=$(clojure -Spath -Sdeps '{:deps {borkdude/missing.test.assertions {:git/url "https://github.com/borkdude/missing.test.assertions" :sha "603cb01bee72fb17addacc53c34c85612684ad70"}}}')

$ lein bb "(require '[missing.test.assertions] '[clojure.test :as t]) (t/deftest foo) (t/run-tests)"

Testing user
WARNING: no assertions made in test foo

Ran 1 tests containing 0 assertions.
0 failures, 0 errors.
{:test 1, :pass 0, :fail 0, :error 0, :type :summary}

medley

A fork of medley made compatible with babashka. Requires bb >= v0.0.58.

clj-http-lite

This fork does not depend on any other libraries. Example:

$ export BABASHKA_CLASSPATH="$(clojure -Sdeps '{:deps {clj-http-lite {:git/url "https://github.com/borkdude/clj-http-lite" :sha "f44ebe45446f0f44f2b73761d102af3da6d0a13e"}}}' -Spath)"

$ bb "(require '[clj-http.lite.client :as client]) (:status (client/get \"https://www.clojure.org\"))"
200

limit-break

A debug REPL library. Example:

$ export BABASHKA_CLASSPATH="$(clojure -Sdeps '{:deps {limit-break {:git/url "https://github.com/technomancy/limit-break" :sha "050fcfa0ea29fe3340927533a6fa6fffe23bfc2f" :deps/manifest :deps}}}' -Spath)"

$ bb "(require '[limit.break :as lb]) (let [x 1] (lb/break))"
Babashka v0.0.49 REPL.
Use :repl/quit or :repl/exit to quit the REPL.
Clojure rocks, Bash reaches.

break> x
1

clojure-csv

A library for reading and writing CSV files. Note that babashka already comes with clojure.data.csv, but in case you need this other library, this is how you can use it:

export BABASHKA_CLASSPATH="$(clojure -Sdeps '{:deps {clojure-csv {:mvn/version "RELEASE"}}}' -Spath)"

./bb -e "
(require '[clojure-csv.core :as csv])
(csv/write-csv (csv/parse-csv \"a,b,c\n1,2,3\"))
"

spartan.test

A minimal test framework compatible with babashka. This library is deprecated since babashka v0.0.68 which has clojure.test built-in.

Blogs

Developing Babashka

Related projects

Gallery

Here's a gallery of more useful examples. Do you have a useful example? PR welcome!

Delete a list of files returned by a Unix command

find . | grep conflict | bb -i '(doseq [f *input*] (.delete (io/file f)))'

Calculate aggregate size of directory

#!/usr/bin/env bb

(as-> (io/file (or (first *command-line-args*) ".")) $
  (file-seq $)
  (map #(.length %) $)
  (reduce + $)
  (/ $ (* 1024 1024))
  (println (str (int $) "M")))
$ dir-size
130M

$ dir-size ~/Dropbox/bin
233M

Shuffle the lines of a file

$ cat /tmp/test.txt
1 Hello
2 Clojure
3 Babashka
4 Goodbye

$ < /tmp/test.txt bb -io '(shuffle *input*)'
3 Babashka
2 Clojure
4 Goodbye
1 Hello

Fetch latest Github release tag

For converting JSON to EDN, see jet.

$ curl -s https://api.github.com/repos/borkdude/babashka/tags |
jet --from json --keywordize --to edn |
bb '(-> *input* first :name (subs 1))'
"0.0.4"

Generate deps.edn entry for a gitlib

#!/usr/bin/env bb

(require '[clojure.java.shell :refer [sh]]
         '[clojure.string :as str])

(let [[username project branch] *command-line-args*
      branch (or branch "master")
      url (str "https://github.com/" username "/" project)
      sha (-> (sh "git" "ls-remote" url branch)
              :out
              (str/split #"\s")
              first)]
  {:git/url url
   :sha sha})
$ gitlib.clj nate fs
{:git/url "https://github.com/nate/fs", :sha "75b9fcd399ac37cb4f9752a4c7a6755f3fbbc000"}
$ clj -Sdeps "{:deps {fs $(gitlib.clj nate fs)}}" \
  -e "(require '[nate.fs :as fs]) (fs/creation-time \".\")"
#object[java.nio.file.attribute.FileTime 0x5c748168 "2019-07-05T14:06:26Z"]

View download statistics from Clojars

Contributed by @plexus.

$ curl https://clojars.org/stats/all.edn |
bb -o '(for [[[group art] counts] *input*] (str (reduce + (vals counts))  " " group "/" art))' |
sort -rn |
less
14113842 clojure-complete/clojure-complete
9065525 clj-time/clj-time
8504122 cheshire/cheshire
...

Portable tree command

See examples/tree.clj.

$ clojure -Sdeps '{:deps {org.clojure/tools.cli {:mvn/version "0.4.2"}}}' examples/tree.clj src
src
└── babashka
    ├── impl
    │   ├── tools
    │   │   └── cli.clj
...

$ examples/tree.clj src
src
└── babashka
    ├── impl
    │   ├── tools
    │   │   └── cli.clj
...

List outdated maven dependencies

See examples/outdated.clj. Inspired by an idea from @seancorfield.

$ cat /tmp/deps.edn
{:deps {cheshire {:mvn/version "5.8.1"}
        clj-http {:mvn/version "3.4.0"}}}

$ examples/outdated.clj /tmp/deps.edn
clj-http/clj-http can be upgraded from 3.4.0 to 3.10.0
cheshire/cheshire can be upgraded from 5.8.1 to 5.9.0

Convert project.clj to deps.edn

Contributed by @plexus.

$ cat project.clj |
sed -e 's/#=//g' -e 's/~@//g' -e 's/~//g' |
bb '(let [{:keys [dependencies source-paths resource-paths]} (apply hash-map (drop 3 *input*))]
  {:paths (into source-paths resource-paths)
   :deps (into {} (for [[d v] dependencies] [d {:mvn/version v}]))}) ' |
jet --pretty > deps.edn

Print current time in California

See examples/pst.clj

Tiny http server

See examples/http_server.clj

Original by @souenzzo

Thanks

License

Copyright © 2019 Michiel Borkent

Distributed under the EPL License. See LICENSE.

This project contains code from:

  • Clojure, which is licensed under the same EPL License.

Can you improve this documentation? These fine people already did:
Michiel Borkent, Arne Brasseur, sogaiu, Peter Nagy, Nikita Prokopov, Peter Strömberg, Gabriel Horner, Nate Sutton & David Harrigan
Edit on GitHub

cljdoc is a website building & hosting documentation for Clojure/Script libraries

× close