The fetched feeds of an app, and keeping them current.
The feeds are a map by key, the guid that the feed was first fetched under. Each entry has the parsed feed, its URL, the ETag and Last-Modified of the last fetch, the guid that the feed gives itself now, when the feed last changed, and the time of the next check:
{"917393e3-…" {:url "https://feeds.podcastindex.org/pc20.xml"
:feed {…}
:retained {item-guid {:missed-at #inst "…"
:gone? true}}
:etag "…"
:last-modified "…"
:guid "917393e3-…" ; what the feed says now
:changed-at #inst "…"
:fetched-at #inst "…"
:next-check #inst "…"
:failures 0
:status 200}}
The feed keeps the items that a fetch no longer brings back, after the fresh ones, since many hosts serve only the newest items. Each has a fact under :retained: when a fetch first missed it, and :gone? when the host took it out rather than leaving it to age out of a capped feed, as it's later than the oldest fresh item, or has no date.
An entry with only its :url is a feed to fetch for the first time, e.g. of a new subscription, and it's due at once. An entry that's :gone? is never due, as default-cadence says, and nor is one that's :local?, the feed of a show of local files, which has no URL.
The fetched feeds of an app, and keeping them current.
The feeds are a map by key, the guid that the feed was first fetched
under. Each entry has the parsed feed, its URL, the ETag and
Last-Modified of the last fetch, the guid that the feed gives itself
now, when the feed last changed, and the time of the next check:
{"917393e3-…" {:url "https://feeds.podcastindex.org/pc20.xml"
:feed {…}
:retained {item-guid {:missed-at #inst "…"
:gone? true}}
:etag "…"
:last-modified "…"
:guid "917393e3-…" ; what the feed says now
:changed-at #inst "…"
:fetched-at #inst "…"
:next-check #inst "…"
:failures 0
:status 200}}
The feed keeps the items that a fetch no longer brings back, after the
fresh ones, since many hosts serve only the newest items. Each has a fact
under :retained: when a fetch first missed it, and :gone? when the host
took it out rather than leaving it to age out of a capped feed, as it's
later than the oldest fresh item, or has no date.
An entry with only its :url is a feed to fetch for the first time, e.g.
of a new subscription, and it's due at once. An entry that's :gone? is
never due, as default-cadence says, and nor is one that's :local?, the
feed of a show of local files, which has no URL.(add feeds url result at)(add feeds url result at cadence)The feeds with the feed in the result of fetching url at at,
under its guid, and its next check by cadence.
The result is that of dk.simongray.podcast/fetch!, and must have a
:feed. An entry under the same guid takes the feed as refresh would,
and the cadence goes over the defaults.
The `feeds` with the feed in the `result` of fetching `url` at `at`, under its guid, and its next check by `cadence`. The `result` is that of dk.simongray.podcast/fetch!, and must have a :feed. An entry under the same guid takes the feed as refresh would, and the `cadence` goes over the defaults.
(affected feeds pings)The guids of the feeds of feeds that the Podping pings name, to
fetch now whatever their next check.
The guids of the feeds of `feeds` that the Podping `pings` name, to fetch now whatever their next check.
(connection-failed? result)Whether the fetch result failed before any server answered: the name
of its host wasn't found, the host couldn't be reached, or the secure
connection failed.
Whether the fetch `result` failed before any server answered: the name of its host wasn't found, the host couldn't be reached, or the secure connection failed.
How many seconds to wait before the next fetch of a feed.
A feed is fetched when it says that it changes, but at most :max-wait later, so that an early or an extra episode shows up the same day. That's twelve hours by default, as in AntennaPod. The wait is :silent for a feed that gives no time itself, :complete for a feed that says it's complete, which :max-wait doesn't cap, and :after-failure for each failure in a row, repeating the last wait.
A feed is gone after its server answers 410 (RFC 9110, section 15.5.11), or after :gone-after-failures failures in a row, about a month at the last wait. It's then fetched only when asked for by name.
How many seconds to wait before the next fetch of a feed. A feed is fetched when it says that it changes, but at most :max-wait later, so that an early or an extra episode shows up the same day. That's twelve hours by default, as in AntennaPod. The wait is :silent for a feed that gives no time itself, :complete for a feed that says it's complete, which :max-wait doesn't cap, and :after-failure for each failure in a row, repeating the last wait. A feed is gone after its server answers 410 (RFC 9110, section 15.5.11), or after :gone-after-failures failures in a row, about a month at the last wait. It's then fetched only when asked for by name.
How many feeds are fetched at once, by default.
How many feeds are fetched at once, by default.
(due feeds at)The guids of the feeds of feeds that are due at at, the earliest
first. A feed that's gone is never due, and nor is one of local files,
which no fetch brings.
The guids of the feeds of `feeds` that are due at `at`, the earliest first. A feed that's gone is never due, and nor is one of local files, which no fetch brings.
(pinged feeds pings)Why the Podping pings name each feed of feeds that they name, by
guid, as the distinct pairs of the :reason and the :medium of its pings,
e.g. {feed-guid #{["live" "podcast"]}} for a live stream that
started.
Why the Podping `pings` name each feed of `feeds` that they name, by
guid, as the distinct pairs of the :reason and the :medium of its pings,
e.g. {feed-guid #{["live" "podcast"]}} for a live stream that
started.(pushed feeds feed-guid feed at)(pushed feeds feed-guid feed at cadence)The feeds after a WebSub hub pushed feed for the feed feed-guid at
at, with cadence, and what it changed, as refresh returns it.
A hub can push only what changed, so a push adds and updates episodes and the feed's details but removes nothing; the next fetch does that. The entry keeps its ETag, Last-Modified and hubs, and must hold its feed.
The `feeds` after a WebSub hub pushed `feed` for the feed `feed-guid` at `at`, with `cadence`, and what it changed, as refresh returns it. A hub can push only what changed, so a push adds and updates episodes and the feed's details but removes nothing; the next fetch does that. The entry keeps its ETag, Last-Modified and hubs, and must hold its feed.
(refresh feeds feed-guid result at)(refresh feeds feed-guid result at cadence)The feeds with result, the result of fetching the feed feed-guid,
applied at at with cadence, and what it changed:
{:feeds {…} ; with the entry of the feed updated
:changed [feed-guid] ; when the fetch gave a fresh feed
:failed [feed-guid] ; when the fetch failed
:moved {feed-guid "https://…"} ; when the feed moved for good
:report [diagnostic]} ; why it failed, and other news
A fresh feed replaces the old one, but for the items that it lacks,
which stay after its own, and a 304 keeps it. A fresh feed that's the
one the entry had, from a server without ETag or Last-Modified, changed
nothing. After a failure, the feed stays and the next check waits longer
for each failure in a row, and the report has :fetch/gone once the feed
is gone, as default-cadence says. A fresh feed without items, where the
old one had some, counts as a failure, with :fetch/empty-feed, since
hosts serve such a feed during an outage. A feed that moved keeps its
guid, and its subscription should move to the URL under :moved. The
:move-report of result, e.g. a move that refresh! couldn't verify,
goes into the report. The cadence goes over the defaults.
The `feeds` with `result`, the result of fetching the feed `feed-guid`,
applied at `at` with `cadence`, and what it changed:
{:feeds {…} ; with the entry of the feed updated
:changed [feed-guid] ; when the fetch gave a fresh feed
:failed [feed-guid] ; when the fetch failed
:moved {feed-guid "https://…"} ; when the feed moved for good
:report [diagnostic]} ; why it failed, and other news
A fresh feed replaces the old one, but for the items that it lacks,
which stay after its own, and a 304 keeps it. A fresh feed that's the
one the entry had, from a server without ETag or Last-Modified, changed
nothing. After a failure, the feed stays and the next check waits longer
for each failure in a row, and the report has :fetch/gone once the feed
is gone, as default-cadence says. A fresh feed without items, where the
old one had some, counts as a failure, with :fetch/empty-feed, since
hosts serve such a feed during an outage. A feed that moved keeps its
guid, and its subscription should move to the URL under :moved. The
:move-report of `result`, e.g. a move that refresh! couldn't verify,
goes into the report. The `cadence` goes over the defaults.(refresh! feeds)(refresh! feeds
{:keys [at feed-guids fetch credentials-of concurrency cadence
max-pages online? max-feeds max-seconds on-feed-fn now-fn]
:or {fetch podcast/fetch!
concurrency default-concurrency
cadence {}
now-fn time/now}
:as opts})Fetch the feeds of feeds that are due with the opts below, and apply
each result.
Returns a map of :feeds, :changed, :failed, :moved and :report for all
the fetches, as refresh gives them, :checked, the guids of the feeds
fetched, and :left, those of the feeds that stay due, in ClojureScript
as a promise. Nothing is thrown for what the network or a server does,
but a platform without an XML parser throws, e.g. a web worker without
a DOMParser installed. The opts are:
The feeds that :max-feeds or :max-seconds leave out stay due, for the next refresh. So do those that the :signal stopped, or kept from starting, with no failure counted, and the report has :refresh/stopped. A round in which every fetch failed before a server answered, at more than one host, counts as offline too. Offline, every feed stays as it was, so that none counts it as a failure and each is due at the next refresh, and the report has :refresh/offline.
A move that a feed declares, e.g. by itunes:new-feed-url, is followed only when a feed of the same show answers at the new URL. Otherwise the feed stays where it is, and the report has :fetch/move-unverified.
Fetch the feeds of `feeds` that are due with the `opts` below, and apply each result. Returns a map of :feeds, :changed, :failed, :moved and :report for all the fetches, as refresh gives them, :checked, the guids of the feeds fetched, and :left, those of the feeds that stay due, in ClojureScript as a promise. Nothing is thrown for what the network or a server does, but a platform without an XML parser throws, e.g. a web worker without a DOMParser installed. The `opts` are: - :at, the instant of the refresh, the time of the clock by default - :feed-guids, the feeds to fetch instead of the due ones, e.g. those that a Podping names, but for those of local files - :fetch, in place of dk.simongray.podcast/fetch! - :credentials-of, a function from the URL of a private feed to its :credentials, so that no password is stored in the feeds and none goes to another URL - :concurrency, how many feeds to fetch at once, default-concurrency by default, and :cadence - :max-pages, the most pages of a paged feed to read at its first fetch, one by default, since the later fetches keep what it read - :online?, false when the device has no network, so that nothing is fetched - :max-feeds, the most feeds to fetch, the earliest due first - :max-seconds, a deadline: no group of fetches starts once that many seconds have passed by the clock of :now-fn, util.time/now by default, so a refresh ends at most a :timeout-seconds later, e.g. within the time that a platform gives a task in the background - :on-feed-fn, a function called as each fetch ends, with a map of the :feed-guid, its :status or the :error, and how many fetches are :done of the :total, e.g. to show progress. On the JVM, fetches that run at once can call it at once. - :signal, to stop the refresh, e.g. when the listener cancels it, as http/send! takes it: an AbortSignal in ClojureScript, and on the JVM an atom that turns true - :send, :timeout-seconds, :max-bytes, :user-agent and :allow-pred, as fetch! takes them The feeds that :max-feeds or :max-seconds leave out stay due, for the next refresh. So do those that the :signal stopped, or kept from starting, with no failure counted, and the report has :refresh/stopped. A round in which every fetch failed before a server answered, at more than one host, counts as offline too. Offline, every feed stays as it was, so that none counts it as a failure and each is due at the next refresh, and the report has :refresh/offline. A move that a feed declares, e.g. by itunes:new-feed-url, is followed only when a feed of the same show answers at the new URL. Otherwise the feed stays where it is, and the report has :fetch/move-unverified.
(slim feed)The part of feed that keeping it current needs: what says when to
check it again, and the guid, date, title and media of each item.
A device that only syncs and refreshes, such as a server, can keep this in place of the whole feed. It gives the same next check, the same new and renamed episodes and the same names for the gpodder API, but no descriptions or the rest. It keeps the key of the feed's metadata, as guid/with-key gives it.
The part of `feed` that keeping it current needs: what says when to check it again, and the guid, date, title and media of each item. A device that only syncs and refreshes, such as a server, can keep this in place of the whole feed. It gives the same next check, the same new and renamed episodes and the same names for the gpodder API, but no descriptions or the rest. It keeps the key of the feed's metadata, as guid/with-key gives it.
cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |