wary-fetch

This is a Clojure and ClojureScript library for fetching URLs that strangers give you, e.g. a feed URL that a user types in, or a link in a document you fetched. Requests and responses are plain maps.

• Only the public internet, if you ask. A stranger's URL can't reach your own machine or your local network, not even by a redirect.
• Limits on every exchange. Neither a slow server nor a gzip bomb can hold you up.
• Secrets that don't leak. Passwords and tokens stay out of your logs, and they don't follow a redirect to another site.
• Polling without waste. A document that hasn't changed costs next to nothing to check, and the response says when to check again.

The same code runs on the JVM, in Node and in the browser, where a proxy on your own server lets a page fetch from sites that don't allow it. There are no dependencies but Clojure.

This library was spun out of podcast-clj, a library for making podcast software, where it fetches feeds, media files and the rest of what a listener subscribes to. Like podcast-clj, it was developed with assistance from frontier LLMs.

Getting started

The library requires Clojure 1.11+ and Java 11+. For the latest release, add it from Clojars to the :deps in your deps.edn:

dk.simongray/wary-fetch {:mvn/version "0.2.0"}

For changes that aren't released yet, use the SHA of the latest commit on master instead:

dk.simongray/wary-fetch
{:git/url "https://github.com/simongray/wary-fetch"
 :git/sha "…"}

For ClojureScript, shadow-cljs only reads Git dependencies from deps.edn, so also set :deps true in your shadow-cljs.edn.

Fetch a document with http/fetch!:

(require '[dk.simongray.wary-fetch :as http])

(def response
  (http/fetch! "https://example.com/"))

(select-keys response [:status :url :last-modified])
;; => {:status        200
;;     :url           "https://example.com/"
;;     :last-modified "Fri, 09 Oct 2026 20:19:47 GMT"}

The response also has the :headers, and the :body as text, decoded by the charset that the server or the document declares. In ClojureScript, http/fetch! gives a promise of the response.

Give the next request the :etag and :last-modified of the response, and a document that hasn't changed comes back as a 304 without a body:

(:status (http/fetch! "https://example.com/"
                      (select-keys response [:etag :last-modified])))
;; => 304

A URL on the network of the app or of its device gets no request, and http/fetch! throws an ex-info with the :type :dk.simongray.wary-fetch/refused instead:

(http/fetch! "http://169.254.169.254/latest/meta-data")
;; throws "The request to http://169.254.169.254/latest/meta-data is not allowed"

The check is the :allow-pred option, a predicate of the URL and of each redirect, which is url/public? unless you give another. To fetch from your own network, give one of your own, e.g. (constantly true).

Requests

A request is a map of :method, :url, :headers and :body. Send one with http/send!, or let http/fetch! make one for you. A request can also have these keys:

• :timeout-seconds, the limit of the whole exchange, 30 by default
• :max-bytes, the limit of the body, 100 MB by default
• :max-redirects, how many redirects to follow, 5 by default
• :allow-pred, a predicate of the URL and of each redirect
• :user-agent, the User-Agent, unless the headers name one
• :as :bytes, for a body of bytes, e.g. of an image
• :range, the first and the last byte to ask for

An exchange that goes beyond a limit throws with the :type :dk.simongray.wary-fetch/timeout or :dk.simongray.wary-fetch/too-large. To tell whether an exception is a timeout, including one from the platform's own client, use http/timeout?.

Use http/send-public! to send a request with url/public? as its :allow-pred, and http/try-send! to get the exception in a map instead of a throw. The http/failure function gives an exception as data.

The functions that make a request for you, e.g. http/fetch!, take the limits, :allow-pred and :user-agent as options, and http/with-options puts them on a request of your own. Unlike http/send!, they take url/public? as the :allow-pred by default. The http/fetch! function also takes :credentials and :accept, and :send for a function that sends the request in place of http/send!.

Credentials

Hide a password or a token with secret/hide, and it prints as #<secret>:

(require '[dk.simongray.wary-fetch.secret :as secret])

(def credentials
  {:username "me" :password (secret/hide "hunter2")})

credentials
;; => {:username "me", :password #<secret>}

(http/fetch! "https://example.com/private.xml" {:credentials credentials})

The Authorization header that the request gets is hidden too. Use http/revealed to get a request with the texts of its hidden headers, e.g. for another HTTP client.

When to ask again

A response from http/fetch! has the :not-before instant if its Retry-After, Cache-Control or Expires header says when to ask again. To get the same instant from the headers of any response, use headers/not-before:

(require '[dk.simongray.wary-fetch.headers :as headers])

(headers/not-before {"cache-control" "max-age=600"}
                    #inst "2026-10-10T12:00:00Z")
;; => #inst "2026-10-10T12:10:00.000-00:00"

The wait is as long as the server says. To poll a server that might be set up wrong, cap the wait with :max-wait-seconds, which both functions take, e.g. 3600 for an hour.

The dk.simongray.wary-fetch.headers namespace has functions for the other headers of a response too, e.g. headers/links for a Link header and headers/media-type for a Content-Type.

Media files

To ask about a large file without downloading it, use http/probe!, which sends a HEAD request and a request for the first two bytes:

(http/probe! "https://upload.wikimedia.org/wikipedia/commons/c/c8/Example.ogg")
;; => {:head {:status 200
;;            :type   "application/ogg"
;;            :length 104793
;;            …}
;;     :get  {:status 206
;;            :range  {:start 0 :end 1 :total 104793}
;;            …}}

Use http/fetcher to read a file in parts by byte range, and, on the JVM, http/digest! to get a SHA-256 or other digest of a file as it streams past.

The proxy

A web page can only read an answer from another site when that site allows it with CORS headers, and most don't. Make a Ring handler with proxy/handler, and mount it on your own server, behind your login, at a route such as /proxy:

(require '[dk.simongray.wary-fetch.proxy :as proxy])

(def proxy-handler
  (proxy/handler))

The handler fetches the URL in its url parameter, as long as that URL is on the public internet, and answers with what came back. In the page, send each request through it with proxy/sender, and the response is what a direct request would give:

(http/fetch! "https://example.com/feed.xml"
             {:send (proxy/sender "/proxy")})

The handler's options, e.g. which headers it passes on, are in proxy/defaults.

In ClojureScript

The functions that send give promises in ClojureScript. To write code for Clojure and ClojureScript at once, chain them with async/then, which takes a value, a future or a promise alike. The dk.simongray.wary-fetch.async namespace has a few more functions like it.

In Node, http/send! sends with node:http, and its guards are those of the JVM. It looks up a name before it asks the :allow-pred, so that url/public? checks every address of the name, and the request then connects only to those addresses. It follows the redirects itself, and asks the :allow-pred about each.

In a browser, http/send! sends with fetch, and two of the guards are weaker:

• A page can't look up a name, so url/public? checks the name itself, and refuses e.g. localhost and names under .local.
• The fetch function follows redirects itself, so :allow-pred doesn't see them.

Those guards matter less in a page, since its requests come from the user's own device rather than from your server. CORS also hides the headers that a server doesn't expose, and :filtered? marks such a response.

Principles

• Standards first. The code follows RFC 9110, RFC 9111 and the other specs it implements, and cites their sections.
• Plain data. Requests and responses are maps, so you can send a request with any HTTP client, or with a function of your own.
• One codebase. Apart from the client of each platform, the library is written in .cljc, with no dependencies but Clojure, and it behaves the same everywhere, as far as JavaScript allows.

Development

clojure -X:test               # the tests on the JVM
npm install                   # once, for the Node tests
clojure -M:cljs compile test  # the tests in Node

The public namespaces are the root, headers, url, proxy, async and secret. The others are internal: jvm, node and browser are the clients of each platform, redirect and limits hold the rules that the clients share, and date, encoding and bytes are helpers.

License

The wary-fetch project is licensed under the MIT licence.