Hello World, With Content Types

In the first two parts of this trail, we made a basic service and we enhanced it to accept a query parameter.

Both of these returned their responses as plain text. Plain text is not particularly useful: it doesn’t excel at presentation the way that HTML does, and it doesn’t offer structured data to a smart client the way JSON (or XML, or EDN, or a number of other formats) would.

In fact, we should go a step further: the client should be in charge of whether they get human-presentable HTML, or code-focused JSON, using the very same endpoints.

Content types are used in HTTP for three purposes:

There’s quite a bit of inherent complexity there, but Pedestal can help sort it all out.

It’s time to see how Pedestal handles content types and response bodies. We will also get our first taste of interceptors.

After reading this guide, you will be able to:

Like hello-world.adoc, this guide is for beginners who are new to Pedestal and may be new to Clojure. It doesn’t assume any prior experience with a Clojure-based web framework. You should be familiar with the basics of HTTP: URLs, response codes, and content types.

If you’ve already done some of those other things, you might want to skip ahead to your-first-api.adoc to start building some logic and multiple routes.

This guide also assumes that you are in a Unix-like development environment, with Java installed. We’ve tested it on Mac OS X and Linux (any flavor) with great results. We haven’t yet tried it on the Windows Subsystem for Linux, but would love to hear from you if you’ve succeeded with it there.

partial$getting-help.adoc

In this guide, we will build on the same hello.clj that we’ve built up over the last two guides. We will enhance it to return a JSON content body. Then we will add the ability to look at the client’s preferred content type and make a decision about what to return.

If you worked through hello-world-query-parameters.adoc then you already have all the files you need. If not, take a moment to grab the sources from the whole shebang in that guide. Feel free to browse the complete sources in the repository, but be warned that the file contains all the versions that we built up through the previous guide. You’ll need to navigate some magic comments to pare it down to just the final version.

Pedestal includes effective support for handilng content type issues on both the request and the response.

In order to appreciate what Pedestal offers, we’ll do it "the hard way", implementing our own support for content types and gradually improving them. This is to show you what goes on under the hood before we use the handy built-in features that Pedestal offers. Don’t worry, this won’t hurt too much. It’ll also let us introduce the most important concept in Pedestal: reference:interceptors.adoc.

Back in hello-world-query-parameters.adoc we echoed an HTTP request back as the response. That was pretty useful for debugging, and it will come in handy for our next step.

Given that we put some effort into getting our /greet route working so nicely, let’s make a new, general route to just echo requests back. Pop open hello.clj and change the routes definition to this:

When you restarted your service, you probably got a nasty message from Clojure like:

That is a very precise way to say we forgot to define echo before using it in the routes definition. Let’s do that now.

We’re going to define echo differently than our handler function greet-handler. Instead of a simple handler function, we’re going to define an interceptor.

reference:interceptors.adoc are the basic unit of work in Pedestal.

At every step of the way between when a request arrives from the client, and when a response is sent back, there’s at least one interceptor. This includes:

An interceptor is essentially a simple map, but the values for the most important keys are not simple values (like strings or numbers), but functions.

This identifies two phases: enter and leave. Each interceptor may participate in the phase by providing corresponding callback functions in the :enter and :leave keys (respectively) ^[1].

Some interceptors only do work during the enter phase, some only during the leave phase, and some do work during both phases. An interceptor that omits the :enter or :leave key will just be skipped — it’s not an error.

But what does that work entail? It’s essentially a bucket brigade: each interceptor is passed the reference:context-map.adoc, is free to perform other work if necessary, and returns a modified context map, which is passed to the next interceptor. An interceptor chain is the process of executing a sequence of interceptors.

Part of the context map is the interceptor queue, and the interceptor stack; the queue is a list of interceptors who have yet to be invoked during the enter phase. As interceptors are invoked^[2], they are also added to the interceptor stack, which is used during the leave phase.

Because it’s a stack, the leave phase invokes the interceptors in the opposite order from the enter phase:

Handler functions, like respond-hello in our example, are special cases. Pedestal can wrap a plain old Clojure function with an interceptor that takes the request map out of the context map, passes it to the function, and uses the return value of the function as the reference:response-map.adoc ^[3].

My advice is to build your value in the context with :enter functions, turn it into a response at the tail end of the interceptor queue, and then refine the response map with defaults, headers, cookies, and so on in :leave functions.

We’re ready to define echo as an interceptor:

We normally wouldn’t write this in such an expanded form, but I wanted to show all the pieces one by one. Ultimately, we’re just making a map with the keys :name and :enter.

We can try that interceptor out now. Bounce your service and use curl to exercise the /echo route:

This should look pretty familiar from the previous guide.

Pedestal’s basic unit of work is the interceptor. The echo and greet-handler functions? Those are actually converted into interceptors. The conn/with-routes in our setup? That creates a routing interceptor and adds it to the list of interceptors in the connector map. The routing interceptor examines the request map and matches it to a route, and pushes the interceptors for the route onto the queue for execution.

Because the interceptor queue is stored in the context, it means any interceptor can modify the queue! There’s nothing that special about the routing interceptor. This is one of the big benefits of using interceptors: the ability to make dynamic decisions during request handling.

Each Pedestal connector implementation may inject some interceptors to the front of the queue, before adding all the interceptors you directly, or indirectly, define in the connector map.

Pedestal connectors are required to allow any of the following types in the :body of the response map:

For each type, the connector will assign a Content-Type header, if not already present, based on the type of value. In our previous attempts with the greet-handler, we returned a String and Pedestal supplied the content type text/plain for us.

The built-in logic does a reasonable job for common cases, but it’s not always right. You might notice that "text/html" doesn’t appear anywhere in that list. We can force that by setting a content type header in our response, like this:

Let’s see the result:

As expected. But, something about that doesn’t seem quite right. We’re completely ignoring HTTP content negotiation. The client might want JSON instead of HTML. Or it might want plain text. Or EDN. The trouble is that the HTTP content negotiation specification is a royal pain. Fortunately, Pedestal provides an interceptor to help.

The api:negotiate-content[ns=io.pedestal.http.content-negotiation] interceptor does the job. If you look at the docs, though, you’ll see that is not an interceptor, but rather a function that returns an interceptor.

This is a common pattern when you need to include some state or customize the behavior of an interceptor. You pass arguments to a function which returns an interceptor that "closes over" those arguments. It returns a data structure that contains functions that carry those arguments around with them.

So what does this interceptor do? It reads the client’s Accept request header (which can have quite a complicated format) and finds the "best match" among the supported content types; this is added to the request map as the key :accept.

If the client doesn’t accept any format that the server supports, then a 406 Not acceptable response is attached to the context.

Let’s remove the content type header from ok and use this new interceptor.

We are using a new namespace here, which will eventually let us write JSON data. That namespace comes from a library that we haven’t included before. So if you try to run this as is, you’ll get an error like this:

That is how Clojure tells you it is missing a library. By looking at the library’s project page, we see that the latest stable release is "2.5.1" (at least, it is when this guide is being written!). We can add the library in the dependencies part of our deps.edn file:

If you try this out, you’ll notice that absolutely nothing changed. That’s because the content negotiation interceptor handles the protocol, but it’s up to you to do something about the result, the :accept key added to the request map.

This gets to the heart of interceptors: breaking up your request processing logic into small, resuable, individually testable chunks.

It’s up to our service code to return a different body format depending on the accepted content type. It probably won’t surprise you that this is a job for another interceptor!

Here’s our first stab at it.

Why does this new interceptor go at the start of the vector? Take look at the picture of the interceptors above. The first one in the vector is called first for the :enter function but last for the :leave function. We want this interceptor to get the last word on the response body so it goes at the top. Turn the queue sideways to write it as a vector, and the topmost interceptor is on the left. It only takes a little bit to get used to this.

Let’s try this out with curl. Restart your service and try sending in some curl requests with different Accept headers:

Notice the content type returned by our service is now "text/html." This is because the curl command, by default, sends an Accept header with the value /. This matches the first value in supported-types, so that’s what’s selected by the content-negotiation-interceptor.

We can use the -H option to override curl 's default. Keep an eye on the "Content-Type" header in each response.

Notice when asking for the response in application/edn format, the string result was enclosed in quotes. That’s the difference between encoding a string as plain text — the string itself — and encoding it in the EDN format, where strings are quoted.

So coerce-body-interceptor works as intended. As usual, I’ve written it in a fairly "non-compact" style so it is easier to see how the parts work with Pedestal. I see a couple of things we can improve though.

First, there’s a straight-up bug. If a previous interceptor has already defined a content type, we should respect that and not overwrite the body or the headers. Second, we’re attaching a new headers map with only the "Content-Type" header. Any other headers attached by other interceptors will be lost. Finally, we could improve the testability by factoring out some of its logic into pure functions.

We’re also going to touch up the echo interceptor while we’re at it.

It is common style in Pedestal applications to make heavy use of the core functions such as clj:update[] and clj:update-in[]. These functions can "reach into" a data structure and make changes to the values already stored there, by applying a function to the stored value. Of course, this is Clojure and the original map isn’t changed, even by update-in: new immutable copies of the nested and containing maps are created.

Using the update style is s more concise than our prior code, and, to a more experienced Clojure developer, often more readable.

Let’s see how this interceptor would change if we refactor it that way.

We’re going to take one more step toward Clojure mastery. Do you see the coerce-body function? It looks inside the context. If there’s already a Content-Type header on the response, then it does nothing and returns the context without modification. On the other hand, if there is no content type assigned yet, it modifies the context by updating the response.

This is a really common pattern in functional languages. You want to make a series of changes to a data structure, where each change is conditional on some other logic. In an imperative language, this would look like a series of if statements whose bodies each mutate the object being built. In Clojure, they would look like a deeply nested if forms, where each if returns either a modified or unmodified version of the input. The trouble with deeply nested `if`s is that it’s too easy to get lost in the nesting.

This pattern is very common, and Clojure includes macros to help. The slightly tricky clj:cond->[]^[4] macro is what we need here. cond-> is similar to the usual threading macro ->, but instead of a just a series of expressions, it precedes each expression with a condition; the condition is evaluated first, and only if true is the value threaded through the expression.

In practice, it’s very straight forward:

If there were more clauses in the cond→, each clause would receive the value returned from the previous clauses.

We’re now getting a fair bit of code. This would be a good time to think about splitting into namespaces for different responsibilities. We’ll tackle that some other time. For now, let’s take a look at the whole thing. Spend some time making sure you understand how and when each line of code gets invoked.

In this guide, we built upon hello-world-query-parameters.adoc to add:

We also learned about interceptors and created a few.

The truth is that server-side applications don’t vend out HTML nearly as much as they once did. APIs are where it’s at. In the next tutorial we will make a REST style API to serve up "TO DO" lists.