Liking cljdoc? Tell your friends :D

pegasus.core


alternate-swap!clj

(alternate-swap! atom f & args)

This version returns the swapped out value as well

This version returns the swapped out value as well
sourceraw docstring

crawlclj

(crawl config)

Main entry point. Right now we have two ways of specifying seed URLs: :seed-list [seed1, seed2, ...] :seed-file /path/to/txt/file/with/seeds. Use just 1 :D.

If you specify a destination (at :destination), all records are finally written there. Otherwise, we pprint to stdout.

Config keys that we understand: :user-agent <user agent>

Main entry point.
Right now we have two ways of specifying seed URLs:
:seed-list [seed1, seed2, ...]
:seed-file /path/to/txt/file/with/seeds.
Use just 1 :D.

If you specify a destination (at :destination), all records
are finally written there. Otherwise, we pprint
to stdout.

Config keys that we understand:
:user-agent <user agent>
sourceraw docstring

enforce-politenessclj

(enforce-politeness config)
source

filter-urisclj

(filter-uris uris config)

Given a list of URIs pick out an unseen distinct set. This is a helper for the enqueue op.

Given a list of URIs pick out an unseen distinct set.
This is a helper for the enqueue op.
sourceraw docstring

get-time-ticksclj

(get-time-ticks n min-delay)

Generates a set of intervals (jodatime objects) separated by min-delay

Generates a set of intervals
(jodatime objects) separated by min-delay
sourceraw docstring

handle-robots-urlclj

(handle-robots-url robots-cache robots-url config)
source

remove-fragmentsclj

(remove-fragments a-uri)
source

start-crawlclj

(start-crawl init-chan config)
source

cljdoc is a website building & hosting documentation for Clojure/Script libraries

× close