scicloj.zulipdata.pull — org.scicloj/zulipdata 0.1.0

Paginated, resumable pulls of Zulip channel history. Each forward window is keyed on (stream-name, anchor-id, batch-size) and cached on disk via pocket — so repeated runs and crash recovery are idempotent.

Cache directory: ZULIP_CACHE_DIR env var, or ~/.cache/zulipdata-clojurians/ by default.

all-messages clj

(all-messages pull-result)

Flatten the :pages result of pull-channel! into a single sequence of messages, de-duplicating by :id (windows are non-overlapping by construction; this is a redundant safety check).

default-batch-size clj

Messages requested per window when pull-channel! is called without an explicit :batch-size. 5000 is also Zulip's per-request cap.

default-parallelism clj

Default number of channels pulled concurrently by pull-channels!. Channels are independent (separate cache keys, separate Zulip endpoints), so per-channel work parallelises cleanly. The cap is small to be respectful of the Zulip API's rate limits.

fetch-window clj

(fetch-window stream-name anchor-id batch-size)

Cached forward window. Returns the deref'd page map.

public-channel-names clj

(public-channel-names)

Names of all channels visible to the bot that are either public or web-public.

pull-channel! clj

(pull-channel! stream-name
               start-anchor-id
               &
               {:keys [batch-size refresh] :or {batch-size default-batch-size}})

Walk forward through stream-name in cached windows, starting at start-anchor-id. Returns {:pages [...], :message-count n}.

Options: :batch-size — messages per window (default 5000) :refresh — when true, any cached page with found_newest: true is invalidated and re-fetched once, then the walk continues if new full windows appeared. Use to catch up after messages were posted since the last pull.

With :refresh false (default), repeated calls are served entirely from cache.

pull-channels! clj

(pull-channels! channel-names
                &
                {:keys [batch-size refresh parallelism]
                 :as opts
                 :or {parallelism default-parallelism}})

Pull a collection of channels by name. Returns a map {channel-name {:pages ... :message-count ... :stream-id ... :first-message-id ...}}.

First-message ids are resolved from /streams. Any unknown channel names are returned under key :not-found as a vector.

Options: :batch-size — passed through to pull-channel! (default 5000) :refresh — passed through to pull-channel! :parallelism — number of channels to pull concurrently (default default-parallelism, currently 8). Pass 1 for fully sequential pulls.

pull-public-channels! clj

(pull-public-channels! & opts)

Convenience: pull every public + web-public channel visible to the bot. Same options as pull-channels!.

web-public-channel-names clj

(web-public-channel-names)

Names of channels visible to the bot that are web-public — readable without logging in. A small subset of public-channel-names, suitable for sharing demo data.