Making Clojure Even Sweeter

Please see the tupelo.core docs further below.

Please see the tupelo.forest docs for further information.

The tupelo-datomic library has been split out into an independent project. Please see the tupelo-datomic project for details.

The standard clojure-csv library has well-tested and useful functions for parsing CSV (Comma Separated Value) text data, but it does not offer all of the convenience one may wish. Tupelo CSV emphasizes the idomatic Clojure usage of data, using sequences and maps. Please see the tupelo.csv docs.

Please see the tupelo.parse docs.

Please see the tupelo.string docs.

Enables type checking in Clojure code via Plumatic Schema. Please see the source code for definitions, and the James Bond example code for examples of the type-checking in action.

Please see the tupelo.types docs.

Please see the tupelo.misc docs.

Please see the tupelo.base64 docs.

Please see the tupelo.base64url docs.

Please see the tupelo.y64 docs.

Have you ever been debugging some code and had trouble printing out intermediate values? For example:

Suppose you want to display the value after the (inc) function. You can’t just insert a (println …​) because the return value of nil will break the pipeline structure. Instead, just use spy:

This tool is named spy since it can display values from inside any threading form without affecting the result of the expression. In this case, spy printed the value 2 resulting from the (inc) expression. Then, the value 2 continued to flow through the following expressions in the pipeline so that the return value of the expression is unchanged.

You can add in a keyword message to label each spy output:

Note that spy works equally well inside either a "thread-first" or a "thread-last" form (e.g. using -> or ->>), without requiring any changes.

How does spy accomplish this trick? The answer is that the keyword message is assumed to be the label, since interesting debug values are more likely to be strings, numbers, or collections like vectors & maps (if both args are keywords, an exception is thrown; use some other technique for debugging this use-case). Thus, spy can detect whether it is in a thread-first or thread-last form, and then label the output correctly. A side benefit is that keywords like :after-inc or just :110 are easy to grep for in output log files.

As a bonus for debugging, the value is output using (pr-str …​) so that numbers and strings are unambiguous in the output:

Sometimes you may prefer to print out the literal expression instead of a keyword label. In this case, just use spyx (short for "spy expression") :

In other instances, you may wish to use spyxx to display the expression, its type, and its value:

Non-pure functions (i.e. those with side-effects) are safe to use with spy. Any expression supplied to spy will be evaluated only once.

Sometimes you may just want to save some repetition for a simple printout:

To be precise, the function signatures for the spy family are:

If you are debugging a series of nested function calls, it can often be handy to indent the spy output to help in visualizing the call sequence. Using with-spy-indent will give you just what you want:

We all love to use the threading macros -> and ->> for certain tasks, but they only work if all of the forms should be threaded into the first or last argument.

The built-in threading macro as-> can avoid this problem, but the order of the first expression and the placeholder symbol is arguably backwards from what most users would expect. Also, there is often no obvious name to use for the placeholder symbol. Re-using a good idea from Groovy, we simply use the symbol it as the placeholder symbol in each expression to represent the value of the previous result.

Here is a more complicated example. Note that we can assign into a local let block from the it placeholder value:

More examples can be found here.

The it-> macro has a cousin cond-it-> that allows you to thread the updated value through both the conditional and the action expressions:

Maps are convenient, especially when keywords are used as functions to look up a value in a map. Unfortunately, attempting to look up a non-existent keyword in a map will return nil. While sometimes convenient, this means that a simple typo in the keyword name will silently return corrupted data (i.e. nil) instead of the desired value.

Instead, use the function grab for keyword/map lookup:

The function grab should also be used in place of clojure.core/get. Simply reverse the order of arguments to match the "keyword-first, map-second" convention.

For looking up values in nested maps, the function fetch-in replaces clojure.core/get-in:

Clojure has functions assoc & assoc-in, update & update-in, and dissoc. However, there is no function dissoc-in. The Tupelo function dissoc-in provides the desired functionality:

The unit test shows the functions in action:

Note that if non-existant keys are included in keys-vec, any missing map layers will be constructed as necessary, which is consistant with the behavior of both clojure.core/assoc-in and clojure.core/update-in (note that nil is the value of the final map entry, not the empty map {} as for the other examples).

Note that only the map entry corresponding to the last key kZ is cleared. This differs from the dissoc-in function in the old clojure-contrib library which had the unpredictable behavior of recursively (& silently) deleting all keys in keys-vec corresponding to empty maps.

The concat function can sometimes have rather surprising results:

In this example, the user probably meant to merge the 3 maps into one. Instead, the three maps were mysteriously converted into length-2 vectors, which were then nested inside another sequence.

The conj function can also surprise the user:

Here the user probably wanted to get [1 2 3 4] back, but instead got a nested vector by mistake.

Instead of having to wonder if the items to be combined will be merged, nested, or converted into another data type, we provide the glue function to always combine like collections together into a result collection of the same type:

An Exception will be thrown if the collections to be 'glued' are not all of the same type. The allowable input types are:

Clojure has the cons, conj, and concat functions, but it is not obvious how they should be used to add a new value to the beginning of a vector or list:

Do you know what conj does when you pass it nil instead of a sequence? It silently replaces it with an empty list: (conj nil 5) ⇒ (5) This can cause you to accumulate items in reverse order if you aren’t aware of the default behavior:

These failures are irritating and unproductive, and the error messages don’t make it obvious what went wrong. Instead, use the simple prepend and append functions to add new elements to the beginning or end of a sequence, respectively:

Both prepend and append always return a vector result.

Suppose we have a mixture of scalars & vectors (or lists) that we want to combine into a single vector. We want a function ??? to give us the following result:

Clojure doesn’t have a function for this. Instead we need to wrap all of the scalars into vectors and then use glue or concat:

It may be inconvenient to always wrap the scalar values into vectors just to combine them with an occasional vector value. Instead, it might be more convenient to unwrap the vector values, then combine the result with other scalars. We can do that with the ->vector and unwrap functions:

It will also work recursively for nested unwrap calls:

Suppose you want to remove an element form a sequence. Did you know that Clojure has no equivalent to Java’s List.remove(int index) function? Well, now it does:

Unlike the raw take and drop functions on which it is based, drop-at will throw an exception for invalid values of index.

Suppose you want to insert an element into a sequence. Tupelo has you covered here as well:

As with assoc, you are allowed to insert the new element into the first empty slot after all existing elements, but no further. insert-at will throw an exception for invalid values of index.

And, of course, you can also replace an element in a sequence:

As with drop-at, replace-at will throw an exception for invalid values of index.

Clojure has an empty? function to indicate if a collection has zero elements or is nil (i.e. not present). However, clojure has no corresponding not-empty? function, and people have written into the mailing wondering where it is. Well, now it is available:

The unit test shows it in action:

Just to confuse things, Clojure does have the similarly named functions empty and not-empty. Be sure to avoid these two functions for predicate tests.

A similar, but more complicated, situation exists in the case of not-any?. Clojure has the not-any? function to indicate if a predicate is false for all items in a collection. However, there has never been a corresponding any? function such that

for any predicate and collection. The situation has become more confusion as of Clojure 1.9.0-alpha10 since a completely unrelated function any? has been added in support of clojure.spec. The new any? function is defined as:

So the new any? function is a semantic mismatch to the not-any? function and is completely unrelated to testing a collection using a predicate.

The Tupelo library attempts to resolve this confusing situation by providing both positive and negative versions of the collection test with a name which does not conflict with either any? or not-any? in clojure.core:

The unit test shows these functions in action:

Sometimes we want an easy way to find out if an item is n a collection. The Tupelo library supplies three convenient functions for this purpose: contains-elem?, contains-key?, and contains-val?.

The most generic function is contains-elem?, which is intended for vectors or any other clojure seq:

Here we see that for an integer range or a mixed vector, contains-elem? works as expected for both existing and non-existant elements in the collection. For maps, we can also search for any key-value pair (expressed as a len-2 vector):

It is also straightforward to search a set:

For maps & sets, it is simpler (& more efficient) to use contains-key? to find a map entry or a set element:

And, for maps, you can also search for values with contains-val?:

As seen in the test, each of these functions works correctly when for searching for nil values.

Clojure’s seq abstraction (and lazy seq’s) is very useful, but sometimes you just want everything to stay in a nice, eager, random-access vector. Here is an eager (non-lazy) version of for which always returns results in a vector:

For most purposes, it is is easy to use (vec some-seq) to convert an arbitrary sequence to a vector. In the event of nested data, we can use (flat-vec …​). This works like flatten but is not lazy and returns results in a nice simple vector.

Clojure training materials seem to vary somewhat in the recommended form for the generation of a lazy sequence. This is further complicated by the legacy function lazy-cat which can easily cause an out-of-memory error (please see this post). A simpler form is possible using tupelo.core/lazy-cons macro. An example of this form in use is:

The new macro lazy-cons accepts the output value as the first arg, and a recursive function call as the second arg. The recursive call will have delayed-execution and will not be invoked until it is required. The (when <condition>) form returns nil to signal the termination of the lazy sequence.

Implementation note:

The canonical structure of when and lazy-cons shown above is not required, but is probably the simplest of multiple possible choices. The new form of (lazy-cons val (recursive-call…​)) is nothing but a simplification of the original clojure.core form (lazy-seq (cons val (recursive-call…​))) which reduces typing and the possibility of errors.

Please note that tupelo.core/lazy-cons bears no relation to the historical lazy-cons which was briefly considered for clojure.core circa 2008.

One of the nice features of Python is the ability to use Generator Functions. These allow a function to "yield" a result from anywhere in the code, which is placed in a lazy output buffer for consumption by the calling function. The generator function is "paused" until the output value is consumed, then resumes execution where it left off with all local state preserved. This ability is especially handy when you have nested loops or other structures that make it inconvenient to return a result as the last expression in a function.

lazy-gen uses a core.async channel to buffer output, with a default buffer size of 32 (controlled by the dynamic var lazy-gen-buffer-size). Result values passed to yield generate a lazy sequence that is the result of the (lazy-gen …​) macro. The closely-related function yield-all inserts the elements of a collection onto the output stream instead of just a single value. Besides doseq, lazy-gen is also very handy for generating a lazy seq within a loop-recur expression.

Within a processing chain, it is often desirable to verify that an intermediate value is within an expected range or of an expected type. The built-in assert function cannot be used for this purpose since it returns nil, and the Plumatic Schema validate can only perform a limited amount of type testing. The (validate …​) function performs arbitrary validation, throwing an exception if a non-truthy result is returned:

A closely related function is verify. It is like validate but accepts an expression instead of a predicate/value pair. Upon success, the expression value is returned; otherwise an exception is thrown:

Sometimes in testing, we want to verify that a key-value pair is present in a map, but we don’t know or care what the value is. For example, Datomic returns maps containing the key :db/id, but the associated value is unpredictable. Tupelo provides the (matches? …​) expression to make these tests a snap:

Note that a wildcard can match either a primitive or a composite value. It works for both maps and vectors. The only restriction is that the wildcard symbol _ (underscore) cannot be used as a key in the pattern-map (it can be used anywhere in a vector-pattern)."

Sometimes using core.match is overkill. For some patterns & values it can run very slowly or even create a stack overflow exception. For most cases, all you really need is a simple wildcard match.

The wild-match? function returns true if a pattern is matched by one or more values. The special keyword :* (colon-star) in the pattern serves as a wildcard value. Note that a wildcard can match either a primitive or a composite value: Usage:

Samples:

Sometimes you want to extract the keys & values from a map for manipulation or extension before building up another map (especially useful for manipulating default function args). Here is very handy function for that:

Sometimes you know an operation may result in an Exception, and you would like to have the Exception converted into a default value. That is when you need:

This feature is put to good use in tupelo.parse, where you will find functions that work like this:

Everyone knows that you shouldn’t compare floating-point numbers (e.g. float, double, etc) for equality since roundoff errors can prevent a precise match between logically equivalent results. However, it has always been awkward to regenerate "approx-equals" code by hand every time new project requires it. Here we have a simple function that compares two floating-point values (cast to double) for relative equality by specifying either the number of significant digits that must match or the maximum error tolerance allowed:

An extract from the unit tests illustrates the use of rel=

Note that, for the :digits variant, 'equality' is truly relative, since only the N most significant digits of each value must match.

Be sure to see the dedicated functions in the tupelo.string namespace!

Suppose you have a bunch of nested results and you just want to convert everything into a single string. In that case, strcat is for you:

Note that any nil values map to the empty string as with clojure.core/str.

Sometimes, you may wish to clip a string to a maximum length for ease of display. In that case, use clip-str:

Notice that clip-str will accept any argument type (map, sequence, etc), and convert it into a string for you. Also, it will work correctly even if the clip-length is an upper bound; shorter strings are returned unchanged.

When processing sequences of data, we often need to extract a sequence of desired data, or, conversely, remove all of the undesired elements. Have you ever been left wondering which of these two forms is correct?

I normally think of filters as removing bad things. Air filters remove dust. Coffee filters keep coffee grounds out of my cup. A noise filter in my stereo removes contaminating frequencies from my music. However, filter in Clojure is written in reverse, so that it keeps items identified by the predicate. Wouldn’t be nicer (and much less ambiguous) if you could just write the following?

It seems to me that keep-if and drop-if are much more natural names and remove ambiguity from the code. Of course, these are just thin wrappers around the built-in clojure.core functions, but they are much less ambiguous. I think they make the code easier to read and the intent more obvious.

The two functions keep-if and drop-if can be used equally well in order to retain or remote elements form a clojure map or set. The semantics for sets look the same as for a sequence (vector or list). The predicate can be any 1-arg function:

Notice that the functions recognized the input collection as a set, and returned a set as the result. Very convenient.

For maps, each element is a MapEntry, which contains both a key and value. keep-if and drop-if understand maps, and will destructure each MapEntry. Thus, the predicate function can be any 2-arg function:

As with sets, the functions recognized that a map was supplied, accepted a 2-arg predicate function, and returned back a map to the user.

Both keep-if and drop-if will throw an Exception if the predicate function supplied has the wrong arity, or if the supplied collection is not one of either the sequential (vector or list), map, or set data types.

The pervasive use of seq’s in Clojure means that scalar values often appear wrapped in a vector or some other sequence type. As a result, one often sees code like (first some-var) and it is not always clear that the code is simply "unwrapping" a scalar value, since there could well be remaining values in the sequence. Indeed, for a length-1 sequence it would be equally valid to use (last some-var) since first=last if there is only one item in the list.

To clarify that we are simply unwrapping a single value from the sequence, we may use the function only:

Clojure has the functions first, second, and requires the use of nth for any subsequent position. Sometimes it is handy to have a quick way to grab the 3rd item from a sequential collection. Tupelo provides the third function to fill this void:

Clojure marries the worlds of Java and Lisp. Unfortunately, these two worlds have different ideas of truth, so Clojure accepts both false and nil as false. Sometimes, however, you want to coerce logical values into literal true or false values, so we provide a simple way to do that:

Since truthy? and falsey? are functions (instead of special forms or macros), we can use them as an argument to filter or any other place that a higher-order-function is required:

Clojure has the build-in function some to return the first truthy value from a sequence argument. It also has the poorly named function some? which returns the value true if a scalar argument satisfies (not (nil? arg)). It is easy to confuse some and some?, not only in their return type but also in the argument they accept (sequence or scalar). In keeping with the style for other basic test functions, we provide the function not-nil? as the opposite of nil?.

The unit tests show how not-nil? leads to a more natural code syntax:

Update 2016-6-13: Now included in clojure.core 1.9.0-alpha5!

In some situations, a function may need to verify that an argument is seqable, that is, will a call to (seq some-arg) succeed? If so, some-arg may be interpreted as a sequence of values. Clojure doesn’t have a built-in function for this (please note that seqable? is different from seq?), but we can copy an solution from the old clojure.contrib.core/seqable: