pegasus.core — cc.artifice/pegasus 0.7.2

alternate-swap! clj

(alternate-swap! atom f & args)

This version returns the swapped out value as well

crawl clj

(crawl config)

Main entry point. Right now we have two ways of specifying seed URLs: :seed-list [seed1, seed2, ...] :seed-file /path/to/txt/file/with/seeds. Use just 1 :D.

If you specify a destination (at :destination), all records are finally written there. Otherwise, we pprint to stdout.

Config keys that we understand: :user-agent <user agent>

enforce-politeness clj

(enforce-politeness config)

filter-uris clj

(filter-uris uris config)

Given a list of URIs pick out an unseen distinct set. This is a helper for the enqueue op.

get-time-ticks clj

(get-time-ticks n min-delay)

Generates a set of intervals (jodatime objects) separated by min-delay

handle-robots-url clj

(handle-robots-url robots-cache robots-url config)

remove-fragments clj

(remove-fragments a-uri)

start-crawl clj

(start-crawl init-chan config)