A breadth-first frontier over plain HTTP, with no crawler library behind it.
It exists because crawler4j (hive-crawl's engine) resolves Tika 1.16 and the ingestor resolves Tika 3: one coordinate, one winner, and in a shared JVM crawler4j's parser dies on a class Tika 2 deleted. Measured, not assumed - see the frontier-clash note in hive memory.
Everything that decides anything here is pure: robots parsing, link
selection, URL normalisation. Only fetch-page touches the network.
A breadth-first frontier over plain HTTP, with no crawler library behind it. It exists because crawler4j (hive-crawl's engine) resolves Tika 1.16 and the ingestor resolves Tika 3: one coordinate, one winner, and in a shared JVM crawler4j's parser dies on a class Tika 2 deleted. Measured, not assumed - see the frontier-clash note in hive memory. Everything that decides anything here is pure: robots parsing, link selection, URL normalisation. Only `fetch-page` touches the network.
(absolute-url base href)HREF resolved against BASE, or nil when it does not resolve to http(s).
HREF resolved against BASE, or nil when it does not resolve to http(s).
(fetch-page http-get url user-agent)GET URL as a string. Returns Result<{:url :html :content-type}>.
GET URL as a string. Returns Result<{:url :html :content-type}>.
(http-frontier)(http-frontier http-get)(http-frontier http-get sleep-fn)A frontier that fetches with clj-http and follows links itself.
HTTP-GET and SLEEP-FN are the injection seam: a test supplies both and the crawl runs with no network and no wall-clock cost.
A frontier that fetches with clj-http and follows links itself. HTTP-GET and SLEEP-FN are the injection seam: a test supplies both and the crawl runs with no network and no wall-clock cost.
(page-links html url link-pattern)(page-links html url link-pattern content-only?)Absolute, de-fragmented, pattern-matching links out of HTML at URL.
CONTENT-ONLY? keeps only the links the extractor also kept as content. A
site's nav is on every page, so following it turns crawl this article and what it cites into a sweep of the whole site - measured: a depth-1 crawl of
one Fowler article reached /boardgames and /videos through the header.
Links from an extractor that predates :in-content? carry no such key and
are kept, so a stale host degrades to the old reach rather than to nothing.
Absolute, de-fragmented, pattern-matching links out of HTML at URL. CONTENT-ONLY? keeps only the links the extractor also kept as content. A site's nav is on every page, so following it turns `crawl this article and what it cites` into a sweep of the whole site - measured: a depth-1 crawl of one Fowler article reached /boardgames and /videos through the header. Links from an extractor that predates `:in-content?` carry no such key and are kept, so a stale host degrades to the old reach rather than to nothing.
(parse-robots text agent)robots.txt TEXT as {:disallow [path] :allow [path] :crawl-delay ms}, for the groups matching AGENT plus the wildcard group.
A deliberately small reading of the standard: prefix rules and Crawl-delay. Anything it cannot parse it ignores, which errs toward fetching - so the politeness delay, which is unconditional, is the real floor.
robots.txt TEXT as {:disallow [path] :allow [path] :crawl-delay ms}, for the
groups matching AGENT plus the wildcard group.
A deliberately small reading of the standard: prefix rules and Crawl-delay.
Anything it cannot parse it ignores, which errs toward fetching - so the
politeness delay, which is unconditional, is the real floor.(robots-allows? rules path)True when RULES permit PATH. The longest matching rule wins, Allow breaking a tie, which is how every major crawler reads it.
True when RULES permit PATH. The longest matching rule wins, Allow breaking a tie, which is how every major crawler reads it.
(robots-url url)The robots.txt URL for URL's origin, or nil.
The robots.txt URL for URL's origin, or nil.
(strip-fragment url)URL without its #fragment. Two links to the same page differing only in fragment are one page, and crawling both doubles the corpus.
URL without its #fragment. Two links to the same page differing only in fragment are one page, and crawling both doubles the corpus.
(textual? content-type)True when a Content-Type is worth parsing. A PDF reached by a link is a job for the ingestor's file path, not for an HTML crawl.
True when a Content-Type is worth parsing. A PDF reached by a link is a job for the ingestor's file path, not for an HTML crawl.
(url-path url)Path (plus query) of URL, defaulting to "/".
Path (plus query) of URL, defaulting to "/".
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |