Turn SOURCES into coerced URPX documents.
A source is anything clojure.java.io/reader coerces: a File, URL, Reader,
InputStream or path string. Where those sources come from is the caller's
business and deliberately not this library's: a corpus repository is a one-off
and there will never be a layout everyone adopts, so walking a tree is
client-specific while turning what it finds into documents is not.
WHY THIS NAMESPACE EXISTS AT ALL. The pipeline it names was previously a
four-step form in the README that every consumer composed by hand, and the
first consumer to do so got it wrong in the one way that does not fail loudly:
mapv where mapcat belongs, and no coercion step, which returned a
plausible number rather than an error. Naming it makes that composition
unavailable to make rather than documented against. Nothing here is new
behavior; it is the same three functions in the one order that is correct.
It lives in its own namespace because it cannot live in either of the two it
spans. urpx.core requires only data.json, java.io and string, and
urpx.coerce requires only malli and urpx.schema; neither requires the
other. Putting this in core would drag Malli into the parse layer, and putting
it in coerce would make a coercion-only consumer pull in the parser.
Turn SOURCES into coerced URPX documents. A source is anything `clojure.java.io/reader` coerces: a File, URL, Reader, InputStream or path string. Where those sources come from is the caller's business and deliberately not this library's: a corpus repository is a one-off and there will never be a layout everyone adopts, so walking a tree is client-specific while turning what it finds into documents is not. WHY THIS NAMESPACE EXISTS AT ALL. The pipeline it names was previously a four-step form in the README that every consumer composed by hand, and the first consumer to do so got it wrong in the one way that does not fail loudly: `mapv` where `mapcat` belongs, and no coercion step, which returned a plausible number rather than an error. Naming it makes that composition unavailable to make rather than documented against. Nothing here is new behavior; it is the same three functions in the one order that is correct. It lives in its own namespace because it cannot live in either of the two it spans. `urpx.core` requires only data.json, java.io and string, and `urpx.coerce` requires only malli and `urpx.schema`; neither requires the other. Putting this in core would drag Malli into the parse layer, and putting it in coerce would make a coercion-only consumer pull in the parser.
A transducer from sources to coerced URPX documents.
(into [] urpx.load/document-xf (my-file-seq))
(eduction urpx.load/document-xf sources) ; realises nothing until pulled
Three steps, each load-bearing, and each one a way to be wrong by hand:
load-graph a document may hold several top-level entities,
so this is mapcat and not map. load-rate-plan
would be wrong here for a different reason: it
throws on a file carrying only a modifier, and
dropping those makes every composition vacuous.
expand-document-entities descends a urpx:URPXPackage wrapper. Without it
a wrapped filing contributes nothing.
coerce-doc strings become java.time values and BigDecimals.
Without it urpx:effectiveDateTime is still a
string, so anything comparing dates throws from
deep inside and anything that does NOT compare
them quietly answers a different question.
ORDER-INDEPENDENT and lazy: the same sources in any order give the same set of
documents, and eduction pulls one source at a time.
coerce-doc returns nil for an entity type this library does not model, so
this DROPS those silently. That is unavoidable in a transducer, which sees one
element at a time, but it is not unobservable: use scan when a caller needs
to assert what was discarded.
A transducer from sources to coerced URPX documents.
(into [] urpx.load/document-xf (my-file-seq))
(eduction urpx.load/document-xf sources) ; realises nothing until pulled
Three steps, each load-bearing, and each one a way to be wrong by hand:
`load-graph` a document may hold several top-level entities,
so this is mapcat and not map. `load-rate-plan`
would be wrong here for a different reason: it
throws on a file carrying only a modifier, and
dropping those makes every composition vacuous.
`expand-document-entities` descends a urpx:URPXPackage wrapper. Without it
a wrapped filing contributes nothing.
`coerce-doc` strings become java.time values and BigDecimals.
Without it urpx:effectiveDateTime is still a
string, so anything comparing dates throws from
deep inside and anything that does NOT compare
them quietly answers a different question.
ORDER-INDEPENDENT and lazy: the same sources in any order give the same set of
documents, and `eduction` pulls one source at a time.
`coerce-doc` returns nil for an entity type this library does not model, so
this DROPS those silently. That is unavoidable in a transducer, which sees one
element at a time, but it is not unobservable: use `scan` when a caller needs
to assert what was discarded.(documents sources)Every coerced URPX document reachable from sources, as a vector.
(into [] document-xf sources), named because that is what callers want.
Every coerced URPX document reachable from `sources`, as a vector. `(into [] document-xf sources)`, named because that is what callers want.
(scan sources)Like documents, but in ONE pass over sources and reporting what was
discarded:
{:urpx.load/documents [<coerced document> ...]
:urpx.load/uncoercible [{:jsonld/type t :jsonld/id id} ...]}
THE DISCARD IS THE POINT. document-xf drops every entity this library does
not model, and over the production corpus that is 42 of 129 expanded entities:
urpx:ProgramEnrollment 30, urpx:SourcePublication 8, urpx:RateCase 4. None is
a version today, which is luck rather than design, and the unlucky form of
exactly this cost a whole rate plan once: a version stating only
urpx:effectiveDate was dropped by a filter that read only
urpx:effectiveDateTime, and nothing said so.
So a caller can assert what it could not before:
(empty? (filter #(str/ends-with? (:jsonld/type %) "Version")
(:urpx.load/uncoercible (scan sources))))
Note that "this library does not model it" and "it does not matter" are different statements. urpx:ProgramEnrollment is 30 of the 42 discards and is modeled nowhere here; whether that matters is the caller's call, and it cannot be made at all while the discard is invisible.
Entries carry @type and @id only, not the entity, so scanning a large
corpus does not retain every graph it walked past.
Like `documents`, but in ONE pass over `sources` and reporting what was
discarded:
{:urpx.load/documents [<coerced document> ...]
:urpx.load/uncoercible [{:jsonld/type t :jsonld/id id} ...]}
THE DISCARD IS THE POINT. `document-xf` drops every entity this library does
not model, and over the production corpus that is 42 of 129 expanded entities:
urpx:ProgramEnrollment 30, urpx:SourcePublication 8, urpx:RateCase 4. None is
a version today, which is luck rather than design, and the unlucky form of
exactly this cost a whole rate plan once: a version stating only
urpx:effectiveDate was dropped by a filter that read only
urpx:effectiveDateTime, and nothing said so.
So a caller can assert what it could not before:
(empty? (filter #(str/ends-with? (:jsonld/type %) "Version")
(:urpx.load/uncoercible (scan sources))))
Note that "this library does not model it" and "it does not matter" are
different statements. urpx:ProgramEnrollment is 30 of the 42 discards and is
modeled nowhere here; whether that matters is the caller's call, and it cannot
be made at all while the discard is invisible.
Entries carry `@type` and `@id` only, not the entity, so scanning a large
corpus does not retain every graph it walked past.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |