IAddon hive.rss. Construction is pure. initialize! validates the config,
loads feed state, registers the rss ingestion source and starts the
scheduler; shutdown! undoes all three.
The scheduler is the addon's own: one daemon thread that wakes every
tick seconds (a quarter hour, or the period when that is shorter) and
polls the feeds that are due. Due-ness is read from the persisted time of
each feed's last poll, so feeds are polled :rss/fetches-per-day times a
day across restarts, never more, and a restart never skips a day.
Config (manifest :addon/config, then config.edn :addons {"hive.rss" {..}}):
:rss/feeds ["https://.." {:feed/url .. :feed/id .. :feed/tags [..]}]
:rss/fetches-per-day 1..96 (default 1)
:rss/project-id memory scope items are filed under (default "hive-rss")
:rss/memory-type entry type (default "note")
:rss/duration entry duration (default "medium")
:rss/sink "auto" | "memory" | "ingestor" (default "auto": through
hive-ingestor when the host offers :tools/invoke)
:rss/max-items-per-poll, :rss/timeout-ms, :rss/max-bytes,
:rss/initial-delay-ms, :rss/state-file
IAddon `hive.rss`. Construction is pure. `initialize!` validates the config,
loads feed state, registers the `rss` ingestion source and starts the
scheduler; `shutdown!` undoes all three.
The scheduler is the addon's own: one daemon thread that wakes every
`tick` seconds (a quarter hour, or the period when that is shorter) and
polls the feeds that are due. Due-ness is read from the persisted time of
each feed's last poll, so feeds are polled `:rss/fetches-per-day` times a
day across restarts, never more, and a restart never skips a day.
Config (manifest `:addon/config`, then config.edn `:addons {"hive.rss" {..}}`):
:rss/feeds ["https://.." {:feed/url .. :feed/id .. :feed/tags [..]}]
:rss/fetches-per-day 1..96 (default 1)
:rss/project-id memory scope items are filed under (default "hive-rss")
:rss/memory-type entry type (default "note")
:rss/duration entry duration (default "medium")
:rss/sink "auto" | "memory" | "ingestor" (default "auto": through
hive-ingestor when the host offers :tools/invoke)
:rss/max-items-per-poll, :rss/timeout-ms, :rss/max-bytes,
:rss/initial-delay-ms, :rss/state-fileFetch one feed over HTTP(S).
Conditional: the ETag and Last-Modified of the previous answer are sent back,
so an unchanged feed costs a 304 and no body. Bounded: a body past
max-bytes is refused while it streams, and every request has a timeout.
Redirects are followed, but never from HTTPS down to HTTP.
Fetch one feed over HTTP(S). Conditional: the ETag and Last-Modified of the previous answer are sent back, so an unchanged feed costs a 304 and no body. Bounded: a body past `max-bytes` is refused while it streams, and every request has a timeout. Redirects are followed, but never from HTTPS down to HTTP.
Deliver new items through hive-ingestor instead of filing notes.
hive-rss never compile-depends on hive-ingestor. Its rss source is
registered through hive-spi, and the ingest is started through the host's
:tools/invoke runtime port as memory ingest source, the same call a
user makes. The ingestor then chunks, embeds and stores each item, writes
its document entry and its KG edges.
Deliver new items through hive-ingestor instead of filing notes. hive-rss never compile-depends on hive-ingestor. Its `rss` source is registered through hive-spi, and the ingest is started through the host's `:tools/invoke` runtime port as `memory ingest source`, the same call a user makes. The ingestor then chunks, embeds and stores each item, writes its document entry and its KG edges.
File entries into hive memory through the hive-spi IMemoryStore port.
The port upserts and nothing more, so this sink does what hive-mcp's
memory add does around it: a content hash, a duplicate check in the
project, the scope:project:<id> tag scope filters match on, and an expiry
from the duration. It never throws; every outcome is a Result.
File entries into hive memory through the hive-spi IMemoryStore port. The port upserts and nothing more, so this sink does what hive-mcp's `memory add` does around it: a content hash, a duplicate check in the project, the `scope:project:<id>` tag scope filters match on, and an expiry from the duration. It never throws; every outcome is a Result.
Per-feed state on disk: seen item keys, validators, and when each feed was last polled. One EDN file, written atomically, readable only by its owner.
A missing or unreadable file is an empty state rather than an error: the worst case is that items already filed are offered again, and the memory sink refuses duplicates by content hash.
Per-feed state on disk: seen item keys, validators, and when each feed was last polled. One EDN file, written atomically, readable only by its owner. A missing or unreadable file is an empty state rather than an error: the worst case is that items already filed are offered again, and the memory sink refuses duplicates by content hash.
Where new items go: filed as memory notes, or ingested by hive-ingestor.
:rss/sink picks. "memory" files notes. "ingestor" ingests, and a
call that fails leaves the items unseen for the next poll. "auto" ingests
when the host offers a :tools/invoke port and falls back to notes for a
batch the ingestor could not take, so items are delivered either way.
Where new items go: filed as memory notes, or ingested by hive-ingestor. `:rss/sink` picks. `"memory"` files notes. `"ingestor"` ingests, and a call that fails leaves the items unseen for the next poll. `"auto"` ingests when the host offers a `:tools/invoke` port and falls back to notes for a batch the ingestor could not take, so items are delivered either way.
One poll of one feed, and one pass over every due feed.
Effects arrive as functions so the pipeline runs the same against the network and the memory store as against test doubles:
:fetch (fn [url {:keys [etag last-modified timeout-ms max-bytes]}]) -> Result :file! (fn [entry]) -> Result {:id :duplicate?} :deliver! (fn [settings subscription feed items]) -> delivery, optional: replaces filing entry by entry (hive-ingestor delivery). A delivery is {:filed [item-key] :created :duplicates :failed :last-error} :save! (fn [state]) persists the whole state :now (fn []) -> epoch seconds
One poll of one feed, and one pass over every due feed.
Effects arrive as functions so the pipeline runs the same against the
network and the memory store as against test doubles:
:fetch (fn [url {:keys [etag last-modified timeout-ms max-bytes]}]) -> Result
:file! (fn [entry]) -> Result {:id :duplicate?}
:deliver! (fn [settings subscription feed items]) -> delivery, optional:
replaces filing entry by entry (hive-ingestor delivery).
A delivery is {:filed [item-key] :created :duplicates :failed :last-error}
:save! (fn [state]) persists the whole state
:now (fn []) -> epoch secondsUser config to validated Settings. No IO: the environment is passed in.
Config arrives flattened from the manifest's :addon/config and the host's
runtime config. A feed may be written as a bare URL or as
{:feed/url .. :feed/id .. :feed/tags [..]}; keys may be keywords or the
strings a JSON config carries ("rss/feeds", "url", "id").
User config to validated Settings. No IO: the environment is passed in.
Config arrives flattened from the manifest's `:addon/config` and the host's
runtime config. A feed may be written as a bare URL or as
`{:feed/url .. :feed/id .. :feed/tags [..]}`; keys may be keywords or the
strings a JSON config carries ("rss/feeds", "url", "id").Feed credentials. Pure: the environment is passed in.
:feed/auth config forms (string keys accepted too):
{:secret-env "HIVE_STORE_KEY"} Basic, key from the environment
{:secret "hv_live_..." :username "me"} Basic, key inline
{:scheme "bearer" :secret-env "FEED_TOKEN"} Bearer
A URL carrying user:secret@ yields a Basic credential and loses the
userinfo. The secret reaches the wire only through authorization.
Feed credentials. Pure: the environment is passed in.
`:feed/auth` config forms (string keys accepted too):
{:secret-env "HIVE_STORE_KEY"} Basic, key from the environment
{:secret "hv_live_..." :username "me"} Basic, key inline
{:scheme "bearer" :secret-env "FEED_TOKEN"} Bearer
A URL carrying `user:secret@` yields a Basic credential and loses the
userinfo. The secret reaches the wire only through `authorization`.A feed item as a hive memory entry, and as an ingestion document. Pure.
A feed item as a hive memory entry, and as an ingestion document. Pure.
Feed XML to data: RSS 2.0, RSS 1.0 (RDF) and Atom 1.0. No IO.
The document is read by the JDK's DOM parser with DOCTYPEs refused, so an entity declaration (billion laughs, external entities) cannot be expanded.
Feed XML to data: RSS 2.0, RSS 1.0 (RDF) and Atom 1.0. No IO. The document is read by the JDK's DOM parser with DOCTYPEs refused, so an entity declaration (billion laughs, external entities) cannot be expanded.
When a feed is due, and which of its items are new. No IO, no clock: the
caller passes now in epoch seconds.
The scheduler does not sleep a whole period between polls. It ticks often
and asks due? of each feed against the time of that feed's last poll, which
is persisted. A restart, a suspended laptop or a slow poll therefore never
skips a day, and never polls a feed more often than the configured rate.
When a feed is due, and which of its items are new. No IO, no clock: the caller passes `now` in epoch seconds. The scheduler does not sleep a whole period between polls. It ticks often and asks `due?` of each feed against the time of that feed's last poll, which is persisted. A restart, a suspended laptop or a slow poll therefore never skips a day, and never polls a feed more often than the configured rate.
The values hive-rss passes between its layers: a subscription, a parsed feed and its items, the schedule, and the per-feed state that survives restarts.
The values hive-rss passes between its layers: a subscription, a parsed feed and its items, the schedule, and the per-feed state that survives restarts.
The rss ingestion source: one document per item of one feed, for
memory ingest source source=rss rss-url=.... The ingest pipeline chunks,
embeds and stores what this returns; the scheduler does not go through it.
The `rss` ingestion source: one document per item of one feed, for `memory ingest source source=rss rss-url=...`. The ingest pipeline chunks, embeds and stores what this returns; the scheduler does not go through it.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |