The web-crawl ISource: a recursive crawl that lands as ingestor Documents.
Stratified as Collect (params -> CrawlSpec), Pipeline (pages -> Documents)
and Boundary (the frontier call). Only fetch-documents reaches the network,
so everything that decides anything is testable without one.
The `web-crawl` ISource: a recursive crawl that lands as ingestor Documents. Stratified as Collect (params -> CrawlSpec), Pipeline (pages -> Documents) and Boundary (the frontier call). Only `fetch-documents` reaches the network, so everything that decides anything is testable without one.
(->spec opts)Promote tool params into a validated CrawlSpec. Returns Result<CrawlSpec>.
:same-domain? defaults to true and only applies when no link pattern was given: an explicit pattern is the caller saying they mean it.
Promote tool params into a validated CrawlSpec. Returns Result<CrawlSpec>. :same-domain? defaults to true and only applies when no link pattern was given: an explicit pattern is the caller saying they mean it.
(as-bool v default)V as a boolean, or DEFAULT when V is absent.
V as a boolean, or DEFAULT when V is absent.
(as-int v default)V as an integer, or DEFAULT. Strings count: MCP numbers arrive as text.
V as an integer, or DEFAULT. Strings count: MCP numbers arrive as text.
(crawled? {:keys [html text]})True when PAGE carries anything worth ingesting.
True when PAGE carries anything worth ingesting.
Defaults for a crawl nobody parameterised.
Deliberately timid: two levels, 25 pages, 300 ms between fetches, robots honoured, and only the links the article itself makes. A default that is polite is one nobody has to remember to be.
Defaults for a crawl nobody parameterised. Deliberately timid: two levels, 25 pages, 300 ms between fetches, robots honoured, and only the links the article itself makes. A default that is polite is one nobody has to remember to be.
(distinct-by-url pages)PAGES with one entry per URL, first occurrence winning.
A crawler reaches the same page by several paths; ingesting it twice puts two copies of the same chunks in front of the embedder.
PAGES with one entry per URL, first occurrence winning. A crawler reaches the same page by several paths; ingesting it twice puts two copies of the same chunks in front of the embedder.
(frontier-for choice)The frontier named by CHOICE.
Defaults to the plain-HTTP frontier: hive-crawl's crawler4j engine resolves
Tika 1.16 while the ingestor resolves Tika 3, so in one JVM crawler4j's
parser dies on a class Tika 2 deleted and every page comes back empty. The
port is the point - naming hive-crawl still selects it where it is viable.
The frontier named by CHOICE. Defaults to the plain-HTTP frontier: hive-crawl's crawler4j engine resolves Tika 1.16 while the ingestor resolves Tika 3, so in one JVM crawler4j's parser dies on a class Tika 2 deleted and every page comes back empty. The port is the point - naming `hive-crawl` still selects it where it is viable.
(page->document {:keys [url html text]} opts)One crawled page as an ingestor Document. Returns Result<Document>.
HTML is preferred over the crawler's text dump: the host extractor turns markup into blocks, headings and fenced code, and none of that is recoverable from an already-flattened page.
One crawled page as an ingestor Document. Returns Result<Document>. HTML is preferred over the crawler's text dump: the host extractor turns markup into blocks, headings and fenced code, and none of that is recoverable from an already-flattened page.
(pages->documents pages opts)PAGES as Documents, one per distinct URL.
Pages that carried nothing, and pages the host could not parse, are dropped rather than failing the crawl: one dead page in fifty is not a failed run.
PAGES as Documents, one per distinct URL. Pages that carried nothing, and pages the host could not parse, are dropped rather than failing the crawl: one dead page in fifty is not a failed run.
(param opts & ks)Read the first of KS present in OPTS, under dashed, underscored and string spellings of each.
Two reasons for the spread. The MCP surface delivers whichever spelling the
caller typed. And the served schema names these params crawl-* - max-depth
is already the ingestor's own corpus-classify depth, and contributing that name
would REDEFINE it - so a call arrives as :crawl-depth while a REPL caller and
this addon's own defaults speak :max-depth.
Presence is decided by contains?, never by truthiness: :same-domain? false
is an answer, and reading it as absence would re-enable the filter it turns off.
Read the first of KS present in OPTS, under dashed, underscored and string spellings of each. Two reasons for the spread. The MCP surface delivers whichever spelling the caller typed. And the served schema names these params `crawl-*` - `max-depth` is already the ingestor's own corpus-classify depth, and contributing that name would REDEFINE it - so a call arrives as :crawl-depth while a REPL caller and this addon's own defaults speak :max-depth. Presence is decided by `contains?`, never by truthiness: `:same-domain? false` is an answer, and reading it as absence would re-enable the filter it turns off.
(same-host-pattern url)A link filter admitting only URLs on the same host as URL.
Returned as a regex STRING because that is what crawler4j's shouldVisit filter takes. Without it a crawl of one article walks the open web.
A link filter admitting only URLs on the same host as URL. Returned as a regex STRING because that is what crawler4j's shouldVisit filter takes. Without it a crawl of one article walks the open web.
(url-host url)Host of URL, or nil when it does not parse.
Host of URL, or nil when it does not parse.
(web-crawl-source)(web-crawl-source opts)Create the web-crawl source.
OPTS may carry :frontier - an ICrawlFrontier to inject, or the name of one. Every other key is remembered as a default and merged under the per-call opts.
Create the `web-crawl` source. OPTS may carry :frontier - an ICrawlFrontier to inject, or the name of one. Every other key is remembered as a default and merged under the per-call opts.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |