hive-ingestor-web.frontier — io.github.hive-agi/hive-ingestor-web 0.1.1

The crawl frontier port, and the hive-crawl adapter behind it.

Depth, politeness and robots.txt are a crawler's job, not an ingestion pipeline's. The pipeline sees only this port; hive-crawl is one adapter and a fake is another, which is what keeps the suite offline.

The adapter resolves hive-crawl at CALL time: hive-crawl is unpublished and deliberately absent from deps.edn, so this namespace must load - and the suite must pass - with it off the classpath.

crawl-output->pages clj

(crawl-output->pages output)

The crawl-site payload as a vector of CrawledPage.

hive-crawl-frontier clj

(hive-crawl-frontier)

(hive-crawl-frontier crawl-fn)

A frontier backed by hive-crawl, resolved lazily.

With CRAWL-FN supplied the resolution is skipped, which is the seam a test or an alternative crawler uses.

ICrawlFrontier clj protocol

crawl-pages clj

(crawl-pages this spec)

Crawl per SPEC (a hive-ingestor-web.schema/CrawlSpec). Returns Result<[CrawledPage]>.

frontier-id clj

(frontier-id this)

Stable id of the frontier implementation.

IFrontierHealth clj protocol

frontier-health clj

(frontier-health this)

Returns {:status :ok|:degraded|:down :details {...}}.

resolve-crawl-site clj

(resolve-crawl-site)

The hive-crawl entry point, or nil when hive-crawl is not on the classpath.

spec->kwargs clj

(spec->kwargs spec)

CrawlSpec as the trailing keyword arguments hive-crawl.core/crawl-site takes.

The crawler4j backend is not a preference: it is the only one of the three that honours robots.txt and a politeness delay, and the only one that surrenders the page HTML rather than a flattened dump.