The corpus — what every reader in this repo converts to, and the one thing that loads it.
A translated ontology is not a dump and not a jar resource: it is a directory of plain vaelii sentence files, one s-expression per line, partitioned by context.
meta.edn format, source, counts, context load order
names.edn foreign term -> vaelii term, with its role
report.edn what converted, what dropped, and why
NOTICE whose knowledge this is, and on what terms
kb/Topology.txt the genlContext wiring of every context
kb/<C>Context.txt that context's `:default`-strength sentences
kb/<C>Context.monotonic.txt its `:monotonic` ones
The NOTICE is not decoration. A translated ontology is a reformulation of
somebody else's knowledge, and every source this repo reads attaches attribution terms
that survive one — OpenCyc says so outright, extending its licence to "renamings and
other logically equivalent reformulations … in any formal language". So the
obligation travels with the corpus and not with this code, which means it has to be
written into the directory: a corpus is data on a filesystem, and the notice is what
keeps it attributable once somebody has copied it somewhere else.
One format, five converters. Cyc, RDF, WordNet, OBO and ATOMIC disagree about
everything except the shape of the answer, so the answer is where they are made to
agree: a corpus says nothing about where it came from beyond meta.edn's :format
line, and load-dir! never asks. That is worth more than tidiness — vaelii's own
catalog recognizes a corpus by meta.edn holding a :context-order and loads it
through one reader, so a directory any of these converters wrote is a directory the
catalog already knows how to open.
The files are read off the filesystem, not the classpath: a converted corpus is data beside a KB directory, not shipped schema, and the interesting ones do not fit in a jar.
load-dir! does not read the files in the order they were written. It reads the
whole corpus five times — term definitions, hierarchy, the rest of the schema, type
memberships, then everything else — and the reason is not speed:
term definitions first is an identity argument. (reifiableFunction F) and
(termOfUnit K E) decide which constant a non-atomic term reifies to, and every
later sentence mentioning that term is reified against them as they stand then.
Arriving late, they mint a second constant for an expression that already has one —
or miss the reification altogether.
genl edges next is a cost argument. An edge retires the memo of the cached
transitive closure and everything else reads that closure back, so interleaved, each
fact pays for a closure the next edge is about to retire.
memberships before facts is a correctness argument. argIsa is open-world
about an argument with no type at all and closed about one that has any, so a fact
checked before its arguments' memberships have arrived is refused on a partial
answer — and the same corpus in a different order would keep a different set of
sentences. Loading every membership first is what makes "what types does this term
have" complete before anything asks.
A refusal is counted, not swallowed: which of vaelii's checks a foreign ontology trips is the most useful thing an import has to say about it.
The corpus — what every reader in this repo converts **to**, and the one thing that
loads it.
A translated ontology is not a dump and not a jar resource: it is a directory of plain
vaelii sentence files, one s-expression per line, partitioned by context.
meta.edn format, source, counts, context load order
names.edn foreign term -> vaelii term, with its role
report.edn what converted, what dropped, and why
NOTICE whose knowledge this is, and on what terms
kb/Topology.txt the genlContext wiring of every context
kb/<C>Context.txt that context's `:default`-strength sentences
kb/<C>Context.monotonic.txt its `:monotonic` ones
**The `NOTICE` is not decoration.** A translated ontology is a *reformulation* of
somebody else's knowledge, and every source this repo reads attaches attribution terms
that survive one — OpenCyc says so outright, extending its licence to "renamings and
other logically equivalent reformulations … in any formal language". So the
obligation travels with the corpus and not with this code, which means it has to be
written into the directory: a corpus is data on a filesystem, and the notice is what
keeps it attributable once somebody has copied it somewhere else.
**One format, five converters.** Cyc, RDF, WordNet, OBO and ATOMIC disagree about
everything except the shape of the answer, so the answer is where they are made to
agree: a corpus says nothing about where it came from beyond `meta.edn`'s `:format`
line, and `load-dir!` never asks. That is worth more than tidiness — vaelii's own
catalog recognizes a corpus by `meta.edn` holding a `:context-order` and loads it
through one reader, so a directory any of these converters wrote is a directory the
catalog already knows how to open.
The files are read off the **filesystem**, not the classpath: a converted corpus is
data beside a KB directory, not shipped schema, and the interesting ones do not fit
in a jar.
## Why loading is layered
`load-dir!` does not read the files in the order they were written. It reads the
whole corpus five times — term definitions, hierarchy, the rest of the schema, type
memberships, then everything else — and the reason is not speed:
* term definitions first is an **identity** argument. `(reifiableFunction F)` and
`(termOfUnit K E)` decide which constant a non-atomic term reifies to, and every
later sentence mentioning that term is reified against them as they stand then.
Arriving late, they mint a second constant for an expression that already has one —
or miss the reification altogether.
* `genl` edges next is a **cost** argument. An edge retires the memo of the cached
transitive closure and everything else reads that closure back, so interleaved, each
fact pays for a closure the next edge is about to retire.
* memberships before facts is a **correctness** argument. `argIsa` is open-world
about an argument with no type at all and closed about one that has any, so a fact
checked before its arguments' memberships have arrived is refused on a partial
answer — and the same corpus in a different order would keep a different set of
sentences. Loading every membership first is what makes "what types does this term
have" complete before anything asks.
A refusal is **counted, not swallowed**: which of vaelii's checks a foreign ontology
trips is the most useful thing an import has to say about it.The four things a counted drop can be, worst last.
A reason name says what was refused; it does not say whether refusing it lost
anything, and summing four unlike things into one :dropped figure is how a report
ends up alarming in the wrong direction. On OpenCyc's OWL export 98.5% of the drops
are :restated — X a owl:Class when the term and its role are already in
names.edn — so a reader that reported only the total would look like it lost a
twentieth of the graph and would be ignored, taking the tenth of a percent that
matters with it.
:restated said elsewhere in the corpus. Nothing is lost, and there is nothing
to fix.:filtered deliberately not imported: an obsolete term, a language nobody asked
for, executable code, an editorial note. A policy, reversible by a
flag wherever one exists.:weakened read in part. The sentence was written; what the source said and
what the corpus holds are not the same claim.:unread a claim with no reading here. The only number that counts against
the reader, and the one to drive down or defend reason by reason.A reason no reader classifies counts as :unread, so a new drop is guilty until its
author says otherwise.
The four things a counted drop can be, worst last.
A reason **name** says what was refused; it does not say whether refusing it lost
anything, and summing four unlike things into one `:dropped` figure is how a report
ends up alarming in the wrong direction. On OpenCyc's OWL export 98.5% of the drops
are `:restated` — `X a owl:Class` when the term and its role are already in
`names.edn` — so a reader that reported only the total would look like it lost a
twentieth of the graph and would be ignored, taking the tenth of a percent that
matters with it.
* `:restated` said elsewhere in the corpus. Nothing is lost, and there is nothing
to fix.
* `:filtered` deliberately not imported: an obsolete term, a language nobody asked
for, executable code, an editorial note. A policy, reversible by a
flag wherever one exists.
* `:weakened` read in part. The sentence *was* written; what the source said and
what the corpus holds are not the same claim.
* `:unread` a claim with no reading here. **The only number that counts against
the reader**, and the one to drive down or defend reason by reason.
A reason no reader classifies counts as `:unread`, so a new drop is guilty until its
author says otherwise.(drop-summary reasons kinds){:drops {kind {reason count}} :unread n} for reasons (a reason -> count map)
under kinds (a reader's reason -> kind table). Kinds with nothing in them are
left out, so a clean conversion's report says so by being short.
`{:drops {kind {reason count}} :unread n}` for `reasons` (a `reason -> count` map)
under `kinds` (a reader's `reason -> kind` table). Kinds with nothing in them are
left out, so a clean conversion's report says so by being short.(general-first edges contexts)The nodes of a [sub super] edge set, most general first — a topological order,
with any cycle broken by name so the result is total.
Contexts load in this order so a context's supercontexts, and the vocabulary its checks
read through them, are in place before its own sentences arrive. The topology file is
written in it too, which is only a readability choice: what an edge costs to assert is
the engine's business, and with-deferred-settle makes it independent of the order
they arrive in.
The nodes of a `[sub super]` edge set, **most general first** — a topological order, with any cycle broken by name so the result is total. Contexts load in this order so a context's supercontexts, and the vocabulary its checks read through them, are in place before its own sentences arrive. The topology file is written in it too, which is only a readability choice: what an edge costs to assert is the engine's business, and `with-deferred-settle` makes it independent of the order they arrive in.
(hierarchy-edge? sentence)A genl / genlContext edge — the schema sentences everything else reads through the
cached closure, so they are their own layer, loaded first.
A `genl` / `genlContext` edge — the schema sentences everything else reads through the cached closure, so they are their own layer, loaded first.
(layer-of sentence)Which of layers a sentence belongs to.
Which of `layers` a sentence belongs to.
The load order. Each layer is read by the one after it — see the namespace docstring for why the order is a correctness claim and not a schedule.
The load order. Each layer is read by the one after it — see the namespace docstring for why the order is a correctness claim and not a schedule.
(load-dir! kb dir)(load-dir! kb dir profiles opts)Load the corpus at dir into kb. Loads Topology.txt first — a context's
supercontexts must exist before its own sentences are checked against them — then each
context file in the meta's topological order, layer by layer (see layers).
profiles is the reader's own map of named subsets; :profile in opts picks one, or
pass :keep-contexts / :drop-contexts / :drop-predicates / :keep-layers
directly. :keep-layers is how a vocabulary is loaded without its instance data:
#{:hierarchy :schema} reads the axioms and stops.
:chain? (default false) is passed to assert: a bulk load derives nothing useful
fact-by-fact, so chaining once at the end (forward-chain) is both faster and the same
fixpoint. In the same process, though — what is owed a derivation is tracked in
memory, not in the store, so a forward-chain against a KB reopened from disk returns
{:derived 0} at once rather than chaining the corpus that is sitting there. Deferring
the chain means deferring it to the end of this call, not to a later session.
:bulk? true runs the engine's bulk-load mode (vaelii.core/*bulk-load?*), which
skips the per-fact definitional checks — the dominant cost is the live (argIsa pred ?n ?type) query every fact pays — and the dedup probe. It stores what a checked load
would have refused, so it is for a corpus a checked load has already reported on,
not for a first look.
:on-progress is called every few thousand sentences with {:phase :done :note};
:note is the context being read. The total is deliberately absent: only the corpus's
own report.edn knows how many sentences it holds, so a caller that wants a percentage
supplies the denominator it read from there. A callback that throws aborts the
load where it stands — that is how a caller cancels one, and the KB is left holding
what had already landed.
Returns {:asserted n :refused n :refusals {reason count} :contexts n}. A refusal is
the engine declining a translated sentence — the :type of the ex-info assert
throws. :on-refusal is handed each one as {:sentence :context :reason :phase :ex},
which is what turns that tally into a diagnosis: a count says a check fired, only the
sentence says whether the ontology disagrees with us or we are wrong. Sampling rather
than keeping every one is the caller's decision, so nothing is retained here.
Named load-dir! rather than load-dir because a failed load leaves the KB partly
populated — the one irreversible thing about it.
Load the corpus at `dir` into `kb`. Loads `Topology.txt` first — a context's
supercontexts must exist before its own sentences are checked against them — then each
context file in the meta's topological order, layer by layer (see `layers`).
`profiles` is the reader's own map of named subsets; `:profile` in `opts` picks one, or
pass `:keep-contexts` / `:drop-contexts` / `:drop-predicates` / `:keep-layers`
directly. `:keep-layers` is how a vocabulary is loaded without its instance data:
`#{:hierarchy :schema}` reads the axioms and stops.
`:chain?` (default false) is passed to `assert`: a bulk load derives nothing useful
fact-by-fact, so chaining once at the end (`forward-chain`) is both faster and the same
fixpoint. **In the same process, though** — what is owed a derivation is tracked in
memory, not in the store, so a `forward-chain` against a KB reopened from disk returns
`{:derived 0}` at once rather than chaining the corpus that is sitting there. Deferring
the chain means deferring it to the end of this call, not to a later session.
`:bulk? true` runs the engine's bulk-load mode (`vaelii.core/*bulk-load?*`), which
skips the per-fact definitional checks — the dominant cost is the live `(argIsa pred ?n
?type)` query every fact pays — and the dedup probe. It stores what a checked load
would have **refused**, so it is for a corpus a checked load has already reported on,
not for a first look.
`:on-progress` is called every few thousand sentences with `{:phase :done :note}`;
`:note` is the context being read. The total is deliberately absent: only the corpus's
own `report.edn` knows how many sentences it holds, so a caller that wants a percentage
supplies the denominator it read from there. A callback that **throws** aborts the
load where it stands — that is how a caller cancels one, and the KB is left holding
what had already landed.
Returns `{:asserted n :refused n :refusals {reason count} :contexts n}`. A refusal is
the engine declining a translated sentence — the `:type` of the `ex-info` `assert`
throws. `:on-refusal` is handed each one as `{:sentence :context :reason :phase :ex}`,
which is what turns that tally into a diagnosis: a count says a check fired, only the
sentence says whether the ontology disagrees with us or we are wrong. Sampling rather
than keeping every one is the caller's decision, so nothing is retained here.
Named `load-dir!` rather than `load-dir` because a failed load leaves the KB partly
populated — the one irreversible thing about it.(read-file-sentences f)Every sentence in KB file f, read with clojure.edn — the corpus is data and can
never run code, exactly as a resources/kb file cannot.
Every sentence in KB file `f`, read with `clojure.edn` — the corpus is data and can never run code, exactly as a `resources/kb` file cannot.
(resolve-profile profiles opts)The load options a :profile name stands for, with any explicit :keep-contexts /
:drop-contexts / :drop-predicates in opts overriding it.
The load options a `:profile` name stands for, with any explicit `:keep-contexts` / `:drop-contexts` / `:drop-predicates` in `opts` overriding it.
The sentences that define the vocabulary rather than state a fact in it: the two
transitive relations, the disjointness and argument constraints, the predicate
metadata, and a function's result declarations. Every one of them changes a cached
closure, or is a declaration the engine reads while storing something else — an
argIsa when a fact is checked, a resultIsa when a NAT is reified. A declaration
that arrives after the content it governs does not apply to it retroactively, so its
layer is not a cost decision.
The sentences that define the vocabulary rather than state a fact in it: the two transitive relations, the disjointness and argument constraints, the predicate metadata, and a function's result declarations. Every one of them **changes a cached closure, or is a declaration the engine reads while storing something else** — an `argIsa` when a fact is checked, a `resultIsa` when a NAT is reified. A declaration that arrives after the content it governs does not apply to it retroactively, so its layer is not a cost decision.
(schema? sentence)Does sentence define the vocabulary rather than state a fact in it?
Does `sentence` define the vocabulary rather than state a fact in it?
(sentence-line sentence)One sentence, printed so clojure.edn/read reads back exactly what was written.
One sentence, printed so `clojure.edn/read` reads back exactly what was written.
(term-definition? sentence)Does sentence define a non-atomic term rather than use one?
Its own layer, ahead of everything, because it decides term identity: a sentence
mentioning (F a) structurally is reified against the declarations as they stand at
that moment. Arriving first, a (termOfUnit K (F a)) is adopted; arriving after, it
mints a second constant for an expression that already has one — same corpus, two
terms, and a collision to reconcile, for no reason but file order. A
reifiableFunction that arrives late is worse still: the NAT was stored as a compound
and no later declaration goes back for it.
Does `sentence` define a non-atomic term rather than use one? Its own layer, ahead of everything, because it decides **term identity**: a sentence mentioning `(F a)` structurally is reified against the declarations as they stand at that moment. Arriving first, a `(termOfUnit K (F a))` is adopted; arriving after, it mints a second constant for an expression that already has one — same corpus, two terms, and a collision to reconcile, for no reason but file order. A `reifiableFunction` that arrives late is worse still: the NAT was stored as a compound and no later declaration goes back for it.
The sentences that say what a non-atomic term is, before anything uses one.
A NAT — (CityInCountryFn Canada), (QuantityFn 5 Meter) — is reified to a constant
as it reaches the store, and these five decide the whole of that: which function
reifies at all, which constant already names an expression, and what a freshly minted
constant is an instance of. Every one is read while storing something else, and
none applies retroactively.
The sentences that say what a **non-atomic term** is, before anything uses one. A NAT — `(CityInCountryFn Canada)`, `(QuantityFn 5 Meter)` — is reified to a constant as it reaches the store, and these five decide the whole of that: which function reifies at all, which constant already names an expression, and what a freshly minted constant is an instance of. Every one is read *while storing something else*, and none applies retroactively.
(type-membership? sentence)A positive unary application — (dog Rover), (binaryPredicate ownerOf) — which is
how vaelii states a type membership, and so is what every argIsa / argGenl /
disjointness check on an ordinary fact reads back.
Structural rather than a functor list, because the corpus's types are the corpus's:
arity 1, and a functor that is a name rather than a frame. not is the only frame a
translation can put in that shape, and (not (dog Rover)) withdraws a membership
rather than stating one, so it is an ordinary fact.
A positive unary application — `(dog Rover)`, `(binaryPredicate ownerOf)` — which is how vaelii states a type membership, and so is what every `argIsa` / `argGenl` / disjointness check on an *ordinary* fact reads back. Structural rather than a functor list, because the corpus's types are the corpus's: arity 1, and a functor that is a name rather than a frame. `not` is the only frame a translation can put in that shape, and `(not (dog Rover))` withdraws a membership rather than stating one, so it is an ordinary fact.
(write! out-dir
{:keys [format source options names root-context notice]}
emit-fn)Write a corpus under out-dir and return its report.
emit-fn is called with one argument, emit! — (fn [context strength sentences]),
where context is a context symbol or :topology for a sentence that belongs to the
context wiring rather than to any one context, and strength is :default or
:monotonic. Whatever emit-fn returns is merged into the report, so a reader says
what it dropped and why in its own vocabulary.
spec is {:format :source :options :names :root-context :notice}.
The root context is where this corpus hangs off vaelii's own vocabulary: it is wired
under CoreContext, and every context the source never placed under another is wired
under it, so no context arrives orphaned however incomplete the source's own hierarchy
was. :notice is the reader's attribution text
for what it just translated — see the namespace docstring for why every corpus gets
one whether or not the reader supplies it.
Streaming: sentences go straight to disk as they are emitted, through a bounded writer pool, so the peak memory is the reader's own working set and not the corpus.
Write a corpus under `out-dir` and return its report.
`emit-fn` is called with one argument, `emit!` — `(fn [context strength sentences])`,
where `context` is a context symbol or `:topology` for a sentence that belongs to the
context wiring rather than to any one context, and `strength` is `:default` or
`:monotonic`. Whatever `emit-fn` returns is merged into the report, so a reader says
what it dropped and why in its own vocabulary.
`spec` is `{:format :source :options :names :root-context :notice}`.
The root context is where this corpus hangs off vaelii's own vocabulary: it is wired
under `CoreContext`, and every context the source never placed under another is wired
under it, so no context arrives orphaned however incomplete the source's own hierarchy
was. `:notice` is the reader's attribution text
for what it just translated — see the namespace docstring for why every corpus gets
one whether or not the reader supplies it.
Streaming: sentences go straight to disk as they are emitted, through a bounded writer
pool, so the peak memory is the reader's own working set and not the corpus.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |