hato

An HTTP client for Clojure, wrapping JDK 11's HttpClient.

It supports both HTTP/1.1 and HTTP/2, with synchronous and asynchronous execution modes.

In general, it will feel familiar to users of http clients like clj-http. The API is designed to be idiomatic and to make common tasks convenient, whilst still allowing the underlying HttpClient to be configured via native Java objects.

Status

hato is under active development. Pre-1.0.0 releases should be considered alpha, and the API is subject to change.

Installation

hato requires JDK 11 and above. If you are running an older vesion of Java, please look at clj-http.

For Leinengen, add this to your project.clj

[hato "0.1.0"]

Quickstart

The main client is available in hato.client.

Require it to get started and make a request:

(ns my.app
  (:require [hato.client :as hc]))
  
  (hc/get "https://httpbin.org/get")
  ; =>
  ; {:request-time 112
  ;  :status 200
  ;  :body "{\"url\" ...}"
  ;  ...}

Usagep

Building a client

Generally, you want to make a reusable client first. This will give you nice things like persistent connections and connection pooling.

This can be done with build-http-client:

; Build the client
(def c (hato/build-http-client {:connect-timeout 10000
                                :follow-redirects :always}))

; Use it for multiple requests
(hc/get "https://httpbin.org/get" {:http-client c})
(hc/head "https://httpbin.org/head" {:http-client c})

build-http-client options

authenticator Used for non-preemptive basic authentication. See the basic-auth request option for pre-emptive authentication. Accepts:

• A map of {:user "username" :pass "password"}
• a java.net.Authenticator

cookie-handler a java.net.CookieHandler if you need full control of your cookies. See cookie-policy for a more convenient option.

cookie-policy Determines whether to accept cookies. The cookie-handler option will take precedence if it is set. If an invalid option is provided, a CookieManager with the default policy (original-server) will be created. Valid options:

• :none Accepts no cookies
• :all Accepts all cookies
• :original-server (default) Accepts cookies from original server
• An implementation of java.net.CookiePolicy.

connect-timeout Timeout to making a connection, in milliseconds (default: unlimited).

redirect-policy Sets the redirect policy.

• :never (default) Never follow redirects.
• :normal Always redirect, except from HTTPS URLs to HTTP URLs.
• :always Always redirect

priority an integer between 1 and 256 (both inclusive) for HTTP/2 requests

proxy Sets a proxy selector. If not set, uses the default system-wide ProxySelector, which can be configured by Java opts such as -Dhttp.proxyHost=somehost and -Dhttp.proxyPort=80 (see all options). Also accepts:

• :no-proxy to explicitly disable the default behavior, implying a direct connection; or
• a java.net.ProxySelector

ssl-context Sets the SSLContext. If not specified, uses the default (SSLContext/getDefault). Accepts:

• a map of :keystore :keystore-pass :trust-store :trust-store-pass. See client authentication examples for more details.
• an javax.net.ssl.SSLContext

ssl-parameters a javax.net.ssl.SSLParameters

version Sets preferred HTTP protocol version.

• :http-1.1 prefer HTTP/1.1
• :http-2 (default) tries to upgrade to HTTP/2, falling back to HTTP/1.1

Making requests

The core function for making requests is hato.client/request, which takes a ring request and returns a response. Convenience wrappers are provided for the http verbs (get, post, put etc.).

; The main request function
(hc/request {:method :get, :url "https://httpbin.org/get"})

; Convenience wrappers
(hc/get "https://httpbin.org/get")
(hc/get "https://httpbin.org/get" {:as :json})
(hc/post "https://httpbin.org/post" {:body "{\"a\": 1}" :content-type :json})

request options

methodLowercase keyword corresponding to a HTTP request method, such as :get or :post.

url An absolute url to the requested resource (e.g. "http://moo.com/api/1").

accept Sets the accept header. a keyword (e.g. :json, for any application/* type) or string (e.g. "text/html") for anything else.

accept-encoding List of string/keywords (e.g. [:gzip]). By default, "gzip, deflate" will be concatenated unless decompress-body? is false.

content-type a keyword (e.g. :json, for any application/* type) or string (e.g. "text/html") for anything else. Sets the appropriate header.

body the body of the request. This should be a string, byte array, input stream, or a java.net.http.HttpRequest$BodyPublisher. Note that clojure data is not automatically coerced to string e.g. sending a json body will require generating a json string via cheshire or other means.

as Return response body in a certain format. Valid options:

• Return an object type: :string (default), :byte-array, :stream, :discarding,
• Coerce response body with certain format: :json, :json-string-keys, :json-strict, :json-strict-string-keys, :clojure, :transit+json, :transit+msgpack. JSON and transit coercion require optional dependencies cheshire and com.cognitect/transit-clj to be installed, respectively.
• A java.net.http.HttpRequest$BodyHandler. Note that decompression is enabled by default but only handled for the options above. A custom BodyHandler may require opting out of compression, or implementing a multimethod specific to the handler.

coerce Determine which status codes to coerce response bodies. :unexceptional (default), :always, :exceptional. This presently only has an effect for json coercions.

query-params A map of options to turn into a query string. See usage examples for details.

multi-param-style Decides how to represent array values when converting query-params into a query string. Accepts:

• When unset (default), a repeating parameter a=1&a=2&a=3
• :array, a repeating param with array suffix: a[]=1&a[]=2&a[]=3
• :index, a repeating param with array suffix and index: a[0]=1&a[1]=2&a[2]=3

headers Map of lower case strings to header values, concatenated with ',' when multiple values for a key. This is presently a slight incompatibility with clj-http, which accepts keyword keys and list values.

basic-auth Performs basic authentication (sending Basic authorization header). Accepts {:user "user" :pass "pass"} Note that basic auth can also be passed via the url (e.g. http://user:pass@moo.com)

oauth-token String, will set Bearer authorization header

decompress-body? By default, sets request header to accept "gzip, deflate" encoding, and decompresses the response. Set to false to turn off this behaviour.

throw-exceptions? By default, the client will throw exceptions for exceptional response statuses. Set this to false to return the response without throwing.

async? Boolean, defaults to false. See below section on async requests.

http-client An HttpClient created by build-http-client or other means. For single-use clients, it also accepts a map of the options accepted by build-http-client.

expect-continue Requests the server to acknowledge the request before sending the body. This is disabled by default.

timeout Timeout to receiving a response, in milliseconds (default: unlimited).

version Sets preferred HTTP protocol version per request.

• :http-1.1 prefer HTTP/1.1
• :http-2 (default) tries to upgrade to HTTP/2, falling back to HTTP/1.1

Usage examples

Async requests

By default, hato performs synchronous requests and directly returns a response map.

By providing async? option to the request, the request will be performed asynchronously, returning a CompletableFuture of the response map. This can be wrapped in e.g. manifold, to give you promise chains etc.

Alternatively, callbacks can be used by passing in respond and raise functions, in which case the CompletableFuture returned can be used to indicate when processing has completed.

; A standard synchronous request
(hc/get "https://httpbin.org/get")

; An async request
(hc/get "https://httpbin.org/get" {:async? true})
; =>
; #object[jdk.internal.net.http.common.MinimalFuture...

; Deref it to get the value87
(-> @(hc/get "https://httpbin.org/get" {:async? true})
    :body)
; =>
; { ...some json body }

; Pass in a callback
(hc/get "https://httpbin.org/get"
       { :async? true } 
       (fn [resp] (println "Got status" (:status resp))) 
       identity)
; =>
; #object[jdk.internal.net.http.common.MinimalFuture...
; Got status 200

(future-done? *1)
; =>
; true

; Exceptional status codes by default will call raise with an ex-info containing the response map.
; This means we can use ex-data to get the data back out. 
@(hc/get "https://httpbin.org/status/400" {:async? true} identity #(-> % ex-data :status))
; =>
; 400

Making queries

hato can generate url encoded query strings in multiple ways

; Via un url
(hc/get "http://moo.com?hello=world&a=1&a=2" {})

; Via query-params
(hc/get "http://moo.com" {:query-params {:hello "world" :a [1 2]}})

; Values are urlencoded
(hc/get "http://moo.com" {:query-params {:q "a-space and-some-chars$&!"}})
; Generates query: "q=a-space+and-some-chars%24%26%21"

; Nested params are flattened by default
(hc/get "http://moo.com" {:query-params {:a {:b {:c 5} :e {:f 6}}}})
; => "a[b][c]=5&a[e][f]=6", url encoded

; Flattening can be disabled
(hc/get "http://moo.com" {:query-params {:a {:b {:c 5} :e {:f 6}}} :ignore-nested-query-string true})
; => "a={:b {:c 5}, :e {:f 6}}", url encoded

Form parameters can also be passed as a map:

(hc/post "http://moo.com" {:form-params {:hello "world"}})

; Nested params are not flattened by default
; Sends a body of "a={:b {:c 5}, :e {:f 6}}", x-www-form-urlencoded
(hc/post "http://moo.com" {:form-params {:a {:b {:c 5} :e {:f 6}}}})

; Flattening can be enabled
; Sends a body of "a[b][c]=5&a[e][f]=6", url encoded
(hc/post "http://moo.com" {:form-params {:a {:b {:c 5} :e {:f 6}}}
                           :flatten-nested-form-params true})

As a convenience, nesting can also be controlled by :flatten-nested-keys:

; Flattens both query and form params
(hc/post "http://moo.com" {... :flatten-nested-keys [:query-params :form-params]})

; Flattens only query params
(hc/post "http://moo.com" {... :flatten-nested-keys [:query-params]})

Output coercion

hato performs output coercion of the response body, returning a string by default.

; Returns a string response
(hc/get "http://moo.com" {})

; Returns a byte array
(hc/get "http://moo.com" {:as :byte-array})

; Returns an InputStream
(hc/get "http://moo.com" {:as :stream})

; Coerces clojure strings
(hc/get "http://moo.com" {:as :clojure})

; Coerces transit. Requires optional dependency com.cognitect/transit-clj.
(hc/get "http://moo.com" {:as :transit+json})
(hc/get "http://moo.com" {:as :transit+msgpack})

; Coerces JSON strings into clojure data structure
; Requires optional dependency cheshire
(hc/get "http://moo.com" {:as :json})
(hc/get "http://moo.com" {:as :json-strict})
(hc/get "http://moo.com" {:as :json-string-keys})
(hc/get "http://moo.com" {:as :json-strict-string-keys})

; Coerce responses with exceptional status codes
(hc/get "http://moo.com" {:as :json :coerce :always})

By default, hato only coerces JSON responses for unexceptional statuses. Control this with the :coerce option:

:unexceptional ; default - only coerce response bodies for unexceptional status codes
:exceptional ; only coerce for exceptional status codes
:always ; coerce for any status code

Certificate authentication

Client authentication can be done by passing in an SSLContext:

; Pass in your credentials
(hc/get "https://secure-url.com" {:http-client {:ssl-context {:keystore (io/resource "somepath.p12") 
                                                              :keystore-pass "password"
                                                              :trust-store (io/resource "cacerts.p12"
                                                              :trust-store-pass "another-password")}}})                                             

; Directly pass in an SSLContext that you made yourself
(hc/get "https://secure-url.com" {:http-client {:ssl-context SomeSSLContext}})

Redirects

By default, hato does not follow redirects. To change this behaviour, use the redirect-policy option.

Implementation notes from the docs:

When automatic redirection occurs, the request method of the redirected request may be modified depending on the specific 30X status code, as specified in RFC 7231. In addition, the 301 and 302 status codes cause a POST request to be converted to a GET in the redirected request.

; Always redirect, except from HTTPS URLs to HTTP URLs
(hc/get "http://moo.com" {:http-client {:redirect-policy :normal}})

; Always redirect
(hc/get "http://moo.com" {:http-client {:redirect-policy :always}})

The Java HttpClient does not provide a direct option for max redirects. By default, it is 5. To change this, set the java option to e.g. -Djdk.httpclient.redirects.retrylimit=10.

The client does not throw an exception if the retry limit has been breached. Instead, it will return a response with the redirect status code (30x) and empty body.

Debugging

To view the logs of the Java client, add the java option -Djdk.httpclient.HttpClient.log=all. In Leinengen, this can be done using :jvm-opts in project.clj.

Other advanced options

# Default keep alive for connection pool is 1200 seconds
-Djdk.httpclient.keepalivetimeout=1200

# Default connection pool size is 0 (unbounded)
-Djdk.httpclient.connectionPoolSize=0

License

Released under the MIT License: http://www.opensource.org/licenses/mit-license.php