itsy.core — itsy 0.1.1

Tool used to crawl web pages with an aritrary handler.

add-worker clj

(add-worker config)

Given a config object, add a worker to the pool, returns the new worker count.

crawl clj

(crawl options)

Crawl a url with the given config.

default-http-opts clj

enqueue-url clj

(enqueue-url config url)

Enqueue the url assuming the url-count is below the limit and we haven't seen this url before.

enqueue-urls clj

(enqueue-urls config urls)

Enqueue a collection of urls for work

extract-all clj

(extract-all original-url body)

Dumb URL extraction based on regular expressions. Extracts relative URLs.

noop clj

(noop & _)

handler that does nothing

palm clj

(palm {:keys [url body]})

It's a HAND-ler.. get it? That was a terrible pun.

remove-worker clj

(remove-worker config)

Given a config object, remove a worker from the pool, returns the new worker count.

start-worker clj

(start-worker config)

Start a worker thread for a config object, updating the config's state with the new Thread object.

stop-workers clj

(stop-workers config)

Given a config object, stop all the workers for that config.

terminated clj

thread-status clj

(thread-status config)

Return a map of threadId to Thread.State for a config object.

url-regex clj

valid-url? clj

(valid-url? url-str)

Test whether a URL is valid, returning a map of information about it if valid, nil otherwise.