Compare two corpora — what one says and the other does not.
Every reader here writes the same corpus, so any two are comparable whatever produced them, and that turns out to be worth more than a debugging aid.
The case that motivated it. OpenCyc publishes its knowledge base twice: as the
binary CFASL dump the inference engine loads, and as an OWL export. This repo reads
both, by two code paths that share nothing below corpus/write! — one is an opcode
table over a byte stream, the other an XML lexer feeding a description-logic
projection. Converting the same KB through both and comparing is the only check
either reader has that is not a fixture somebody wrote to match what the code already
did. If the taxonomies agree, two independent implementations agree; where they
differ, the difference is a finding — about a reader, or about the two releases.
Because they are two releases. The OWL export and the CFASL dump ship on their own schedules, so the diff is not expected to be empty and an empty one would be the surprising result. What the diff is for is telling the two kinds of difference apart: a term missing on one side because the release genuinely lacks it, and a term missing because a reader dropped it.
What is compared, and why not everything. The taxonomy — genl edges and the
terms in them — plus a per-predicate sentence count. Not the full sentence sets: a
corpus of two million sentences would need a real graph diff to compare completely,
and most of the difference would be lexical noise (labels, comments, identifiers) that
says nothing about whether the two agree. The taxonomy is the part with a truth
value.
Names are the join key, and that is sound here for a specific reason: both readers
name from the same underlying Cyc term names, so #$BadmintonAccessory in the dump
and Badminton_accessory in the OWL both spell to badminton_accessory. Comparing
corpora from unrelated sources would need a mapping and this offers none.
Compare two corpora — what one says and the other does not. Every reader here writes the same corpus, so any two are comparable whatever produced them, and that turns out to be worth more than a debugging aid. **The case that motivated it.** OpenCyc publishes its knowledge base twice: as the binary CFASL dump the inference engine loads, and as an OWL export. This repo reads both, by two code paths that share nothing below `corpus/write!` — one is an opcode table over a byte stream, the other an XML lexer feeding a description-logic projection. Converting the same KB through both and comparing is the only check either reader has that is not a fixture somebody wrote to match what the code already did. If the taxonomies agree, two independent implementations agree; where they differ, the difference is a *finding* — about a reader, or about the two releases. Because they are two releases. The OWL export and the CFASL dump ship on their own schedules, so the diff is not expected to be empty and an empty one would be the surprising result. What the diff is for is telling the two kinds of difference apart: a term missing on one side because the release genuinely lacks it, and a term missing because a reader dropped it. **What is compared, and why not everything.** The *taxonomy* — `genl` edges and the terms in them — plus a per-predicate sentence count. Not the full sentence sets: a corpus of two million sentences would need a real graph diff to compare completely, and most of the difference would be lexical noise (labels, comments, identifiers) that says nothing about whether the two agree. The taxonomy is the part with a truth value. Names are the join key, and that is sound here for a specific reason: both readers name from the same underlying Cyc term names, so `#$BadmintonAccessory` in the dump and `Badminton_accessory` in the OWL both spell to `badminton_accessory`. Comparing corpora from unrelated sources would need a mapping and this offers none.
(compare-corpora a b)(compare-corpora a b {:keys [sample] :or {sample 40}})(summarize a) against (summarize b), as a report.
Each side's exclusives are capped at :sample (default 40) in :examples, because the
count is the finding and forty names are enough to see what kind of thing is on one
side only. The counts themselves are never truncated.
`(summarize a)` against `(summarize b)`, as a report. Each side's exclusives are capped at `:sample` (default 40) in `:examples`, because the count is the finding and forty names are enough to see *what kind* of thing is on one side only. The counts themselves are never truncated.
(print-report {:keys [a b shared only-in-a only-in-b predicates examples]})The comparison as something to read, rather than as edn to grep.
The comparison as something to read, rather than as edn to grep.
(summarize dir)The comparable shape of the corpus at dir:
{:sentences n :contexts n :predicates {p n} :genl #{[child parent]} :terms #{t}}
Streamed a file at a time, because the corpora this exists for hold millions of sentences and the point is to compare them, not to hold two of them at once.
The comparable shape of the corpus at `dir`:
{:sentences n :contexts n :predicates {p n} :genl #{[child parent]} :terms #{t}}
Streamed a file at a time, because the corpora this exists for hold millions of
sentences and the point is to compare them, not to hold two of them at once.cljdoc builds & hosts documentation for Clojure/Script libraries
| Ctrl+k | Jump to recent docs |
| ← | Move to previous article |
| → | Move to next article |
| Ctrl+/ | Jump to the search field |