IAddon that teaches hive-ingestor about the RFC series.
The host owns the seams; this addon owns the corpus. On initialize! it
registers its ISource factory in the host's source registry, so the series
becomes reachable as ingest source source=rfc dir=... without the host
ever naming an RFC. shutdown! retracts every registration it owns.
IAddon that teaches hive-ingestor about the RFC series. The host owns the seams; this addon owns the corpus. On initialize! it registers its ISource factory in the host's source registry, so the series becomes reachable as `ingest source source=rfc dir=...` without the host ever naming an RFC. shutdown! retracts every registration it owns.
The RFC registry: what a document cannot know about itself.
An RFC's masthead carries Obsoletes:/Updates: — edges pointing BACKWARD. Whether the document has since been obsoleted, and what its current status is, exist only in the registry (rfc-index.xml). A parser reading the .txt can build supersession edges but can never mark a document HISTORIC, so retrieval over the corpus answers present-tense questions confidently and wrongly.
This namespace turns the registry into facets the ingest pipeline attaches to both the document and its chunks.
The RFC registry: what a document cannot know about itself. An RFC's masthead carries Obsoletes:/Updates: — edges pointing BACKWARD. Whether the document has since been obsoleted, and what its current status is, exist only in the registry (rfc-index.xml). A parser reading the .txt can build supersession edges but can never mark a document HISTORIC, so retrieval over the corpus answers present-tense questions confidently and wrongly. This namespace turns the registry into facets the ingest pipeline attaches to both the document and its chunks.
ISource over a local rsync mirror of the RFC series.
Bulk access to the series is rsync, not HTTP: rsync -avz --delete rsync.rfc-editor.org::rfcs-text-only <dir>
Only the transport differs from the web source — the same rule chain, the same RfcTextParser and the same Document shape are reused through web-docs/body->document, and :document/source is the canonical rfc-editor URL, so document ids and cross-document KG edges agree with anything ingested over HTTP.
The series is ~10k files; a run is a WINDOW over it (:from, :to, :offset, :limit). Re-running a window is an upsert, not a duplicate, because chunk and document ids are derived from the source URL — that is the resumable story: crash, re-run the window, continue.
ISource over a local rsync mirror of the RFC series. Bulk access to the series is rsync, not HTTP: rsync -avz --delete rsync.rfc-editor.org::rfcs-text-only <dir> Only the transport differs from the web source — the same rule chain, the same RfcTextParser and the same Document shape are reused through web-docs/body->document, and :document/source is the canonical rfc-editor URL, so document ids and cross-document KG edges agree with anything ingested over HTTP. The series is ~10k files; a run is a WINDOW over it (:from, :to, :offset, :limit). Re-running a window is an upsert, not a duplicate, because chunk and document ids are derived from the source URL — that is the resumable story: crash, re-run the window, continue.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |