darkstar.watch — com.github.brettatoms/darkstar 0.1.21

Subscriptions discovered from the render, rather than declared beside it.

The problem this solves

A component that updates when one thing changes previously had to say so three times, and keep the three consistent by hand:

:mount     (fn [ctx] {:online? (presence/online? ch user)})  ; depends on…
:subscribe (fn [ctx] [[:presence ch user]])                  ; …watch this…
:render    (fn [view] (boundary [] …))                       ; …patch here

Three declarations of one fact. Every live-children bug in this project was one of the three drifting from the other two, which is a structural problem rather than a run of bad luck.

watch collapses them. It reads a value and, as a side effect of rendering, records that the surrounding fragment depended on that topic:

(fragment (member-id username)
  (fn []
    (let [online? (watch [:presence ch username]
                         #(presence/online? ch username))]
      [:li {:id (member-id username)} …])))

The subscription set is now whatever the render actually read, so it cannot disagree with the render. There is one site to change.

The subscription set changes, so re-derive it on every render

This follows from the above but is easy to miss, and missing it produces a bug that looks like a pass. A dependency set is data: a roster that reads one presence topic per member depends on a different set of topics the moment its membership changes. Subscribing once, at mount, is therefore not enough.

Observed in a browser: a viewer mounted alone, a second member joined, and the viewer's roster re-rendered and showed them correctly — but that viewer had never subscribed to the new member's presence topic, because it did not exist at mount. The joiner appeared and then never went offline. The first render and the join both looked right, which is what let it through a headless test.

So a caller must treat the topics from every render as the current truth. See darkstar.live/refresh!, which reports :added and :removed for exactly this.

The laziness hazard — read this before using it

Recording happens in a dynamic binding, so anything evaluated outside that binding is invisible to it. A lazy sequence escapes:

[:ul (for [m members] (member-row m))]    ; WRONG — rows render later
[:ul (mapv member-row members)]           ; right — forced inside

This fails silently: the first render is correct because the seq is realised during serialisation, but the rows' topics were never recorded, so nothing ever re-renders them. render-recording therefore walks its result and throws on an unrealised lazy seq, which turns a silent staleness bug into a loud one.

Relationship to :subscribe and boundaries

This namespace requires nothing, and it is optional. A darkstar application can push patches by hand — naming a selector and rendering the HTML for it — and never touch watch at all. The two styles also mix within one component: markup wrapped in fragment is managed by darkstar.live, and plain hiccup beside it is the application's to push. Verified, not assumed.

So the trade is worth stating plainly. Manual pushes fail loudly — a wrong selector produces PatchElementsNoTargetsFound in the browser console. watch fails silently, and darkstar.diagnose exists because of that.

Why "fragment"

It was region first, which was wrong on two counts. It described geometry — an area of the screen — when the interesting property is being independently pushable; a plain [:div] is also an area of the screen. And it did not pair with watch: a verb about dependency beside a noun about layout gave no hint the two are halves of one mechanism.

"Fragment" is what this already is on the wire, and the word Datastar and htmx users have for a piece of HTML pushed on its own. watch says why it re-renders; fragment says what gets sent.

fragment clj

(fragment id body-fn)

(fragment id {:keys [static?] :as _opts} body-fn)

Marks body-fn's output as a patchable fragment identified by id, recording the topics its body read.

The rendered root element must carry {:id id}. That is checked here, not left to the author — see assert-id! for why it is verified rather than generated. The usual shape is one *-id function called from both sites:

(defn member-id [u] (str "member-" u))

(fragment (member-id username)
  (fn [] [:li {:id (member-id username)} …]))

The body must read at least one topic through watch, or contain a fragment that does. A fragment with no dependencies can never be updated, which is a silent freeze rather than an error — see assert-subscribed!. For markup that is deliberately static, declare it:

(fragment "title" {:static? true}
  (fn [] [:h1 {:id "title"} "Builds"]))

Nested fragments are kept: an inner fragment is recorded in its own right and its topics bubble up, so a hint reaches both the narrow fragment and any fragment containing it. A caller that patches the narrowest match gets minimal updates; one that patches the outermost still gets correctness.

fragments-for-topic clj

(fragments-for-topic fragments topic)

Region ids whose recorded topics include topic.

A hint names a topic; this answers which fragments have to re-render. Returned narrowest-first — by number of topics read, ascending — so a caller that wants minimal updates can take the first and a caller that wants safety can take them all.

innermost-independent clj

(innermost-independent fragments ids)

Of the fragment ids in ids, those that do not contain another id in ids.

Why this is not just "the narrowest match"

A hint can dirty several fragments at once, and they fall into two very different relationships:

• ancestor/descendant. A containing fragment inherits its children's topics, so a hint for a child also matches the parent. Patching both sends the change twice and the outer patch replaces the inner element that was just updated — the bug the old engine showed as duplicated list items.
• siblings. Two unrelated fragments can read the same topic. A roster and a message list both reading [:channel id] is the ordinary case, not a corner one. Here BOTH must be patched.

Taking the single narrowest match handles the first and silently breaks the second: the sibling that loses the tie never updates. Found in a browser, where a member joining left the other viewer's roster stale because messages and roster had the same topic count and messages sorted first.

So the rule is containment, not count: drop a dirty fragment when a dirty descendant of it is also present, and keep everything else. The survivors are pairwise independent — no one contains another — so patching all of them sends each change exactly once.

re-render clj

(re-render {:keys [body-fn static?] :as _fragment} id)

Re-renders a single recorded fragment, returning {:tree :topics :reads}.

This is the other half of the performance story, and the more structural half. A presence change to one roster row should cost one row, but refresh! originally re-rendered the whole component to obtain it — so a 500-member roster paid 500 rows to update one.

A fragment's body-fn is a closure over exactly what it needs, so it can be called on its own. Recording during that call keeps its topics and reads current, which is what lets a fragment's dependencies change over time without a full render.

Nested fragments inside body-fn are re-recorded and returned in :fragments, so a caller can refresh its stored map for the subtree it just rendered.

The subscription check runs here too, carrying :static? over from the recorded entry. A fragment's dependencies are data and can change between renders, so one that read a topic at mount can stop reading on a later pass — the same freeze, arriving later.

recording? clj

(recording?)

True inside a recording render.

Public so a caller can tell whether watch will record or merely read — useful when a component is rendered outside the engine, in a test or at a REPL.

render-recording clj

(render-recording render-fn)

Renders render-fn and returns {:tree :topics :fragments :reads}.

• :topics every topic read anywhere, which is the component's subscription set
• :fragments {id {:topics :tree :reads :body-fn}}, one entry per fragment
• :reads {topic value} for every read, for unchanged?
• :tree the rendered output, unchanged

Throws on an unrealised lazy seq — see assert-realised!.

subscriptions clj

(subscriptions {:keys [topics]})

The topic set from a recording, shaped for pubsub/subscribe-context!.

This is what replaces a hand-written :subscribe: the topics are the ones the render read, so they are correct by construction.

unchanged? clj

(unchanged? reads)

(unchanged? reads read-fns)

Whether every topic in reads still returns what it returned before.

reads is {topic [{:value v :read-fn f} ...]} from a previous render — a vector because one fragment may read one topic several times. True means no fragment depending on these topics can have changed, so nothing needs rendering.

Optional read-fns is {topic (fn [] v)}, overriding the recorded reader for that topic. Rarely needed — it exists for a caller reconstructing state from outside a render.

Why this is the whole optimisation

A fragment's output is a pure function of the values it read through watch. So if every read is unchanged, the HTML is unchanged — without producing it. That turns a no-op hint from a full component render into a handful of pointer comparisons.

It matters because hints carry no payload and are meant to be published liberally; a design that punishes speculative hints undermines its own transport. Measured on a 500-member roster: re-rendering to discover a no-op cost 45.9ms across 50 viewers, against 0.01ms for the old engine's identical? pruning. This closes that gap rather than accepting it.

The reader comes from the render

watch records the read-fn the component passed it, so re-reading uses the same function the component used. It therefore cannot disagree with the component — which an application-supplied table of readers could, and did: that table was a second place to state every dependency, working directly against the one thing watch exists to fix.

The stale-closure hazard

A recorded reader must re-read current state when called. That is what the natural form does:

(watch [:presence u] #(online? ch u))          ; re-reads — correct
(watch [:presence u] #(contains? @presence u)) ; derefs inside — correct

A reader closing over an already-dereferenced value does not:

(let [snap @presence]
  (watch [:presence u] #(contains? snap u)))   ; WRONG — frozen at first render

That form reports "unchanged" forever and the fragment never updates again. It is the dangerous direction: a false positive, not a false negative. The safe forms are what one writes without thinking about it, and every watch call in the example apps is one of them — but the failure is silent, so it is worth knowing.

False negatives remain possible and harmless: a collection rebuilt on each read compares as changed and re-renders.

watch clj

(watch topic read-fn)

Reads (read-fn), recording that the surrounding fragment depends on topic.

Outside a recording render this is exactly (read-fn), so a component stays callable on its own — which is what keeps it testable without an engine.