vaelii.foreign.wordnet — com.vaelii/vaelii-foreign 0.1.0

Read a WordNet database — Princeton's own dict/ directory — and translate it into a vaelii corpus.

WordNet is the odd one out here: it is a lexical database, organized around which words mean the same thing, and only incidentally an ontology. What makes it worth importing anyway is that its central relation is subsumption. A synset is a set of synonymous words, @ points from a synset to a more general one, and 82,000 noun synsets wired that way are a usable upper taxonomy that nobody has to build.

The format

data.noun, data.verb, data.adj and data.adv — the WNDB format, which has not changed since 1993 and which Open English WordNet still publishes alongside its XML:

02084071 05 n 01 dog 0 019 @ 02083346 n 0000 ~ 02084732 n 0000 … | a member of the genus Canis

offset  lexfile  pos  w_cnt(hex)  word lex_id…  p_cnt  pointers…  | gloss

A pointer is symbol offset pos source/target. This reader takes the data files alone: they are self-contained, and the index files exist to answer "which senses does this word have", which is a lookup rather than a fact.

What comes across

WordNet	vaelii
a synset	a type — dog_n, spelled from its first word
@ hypernym	(genl dog_n canine_n)
@i instance hypernym	the synset is an individual: (composer_n Bach)
& similar to (satellite)	(genl damp_a wet_a)
* entailment	a defeasible rule — see below
%m %s %p meronyms	(memberMeronym …), (substanceMeronym …), (partMeronym …)
! = > ^ $ < \\ +	(antonym …), (attribute …), (causes …), …
;c ;r ;u domains	(topicDomain …), (regionDomain …), (usageDomain …)
each word of a synset	(wordForm dog_n "dog")
the gloss	(comment dog_n "a member of the genus Canis")

~ and the holonyms are not dropped for being uninteresting — they are the exact inverses of @ and the meronyms, stored in both directions because WordNet is a lookup structure. Importing both would double the corpus to say each thing twice.

Entailment becomes a rule. snore *> sleep says that snoring entails sleeping, and a verb synset is a type of event, so this reads as (implies (snore_v ?x) (sleep_v ?x)) over events — the one relation in WordNet with a genuine inferential reading, and it is why importing this into a rule engine is worth doing at all. It lands defeasible (set/defaultRule), because it is a generalization about usage rather than a definition, and it fires on nothing WordNet itself contains: it is vocabulary for somebody else's events. :entailment-rules? false writes plain (entails a b) facts instead.

Naming and contexts

A synset is named from its first word plus its part of speech — dog_n, run_v — which collides constantly, because that is what a word having several senses is. Collisions take a numeric suffix in offset order, so dog_n and dog_n_2 are stable across runs but are not WordNet's own sense numbers, which live in the index files this reader does not read. (wnOffset dog_n "n02084071") is written for every synset, and is the identifier to join on.

Each part of speech is its own context under WordNetContext, so a profile can load the noun taxonomy without the other three.

classify clj

(classify synsets)

Pass 1. {:entries [[key role base] …]} — one entry per synset, in file and offset order.

A synset with an instance hypernym is an individual, not a type: WordNet marks Bach as an instance of composer rather than a kind of one, and that distinction is exactly vaelii's between (genl a b) and (b A). Everything else is a type.

convert! clj

(convert! path out-dir)

(convert! path out-dir opts)

Convert the WordNet database at path — a dict/ directory, or one data.<pos> file — into a vaelii corpus under out-dir. Two passes: the first names every synset, the second writes the sentences. Returns the report map.

Options: :entailment-rules? writes verb entailment as a defeasible rule (on — see the namespace docstring); :word-forms? writes a wordForm fact per word of a synset (on); :limit reads only the first n synsets.

drop-flags clj

The convert option that keeps each :filtered drop — see cyc/drop-flags for the contract and plugin-test for what enforces it.

Empty, and it should be: this reader's one drop reason is :restated, not :filtered. ~ is @ stored backwards and importing both would double the corpus to say each thing twice, which is not a policy anybody would want reversed.

drop-kinds clj

What each of this reader's drop reasons is — see corpus/drop-kinds. A reason missing here counts as :unread.

:inverse-pointer is by far the largest and is :restated outright: WordNet is a lookup structure and stores ~ beside @, so importing both would double the corpus to say each thing twice. It is a drop only in the sense that a line of the file produced no sentence.

:malformed-record is deliberately not in here: a record whose own counts do not line up is content this reader could not carry, and :unread is what that is.

load-dir! clj

(load-dir! kb dir)

(load-dir! kb dir opts)

Load the corpus at dir into kb — vaelii.foreign.corpus/load-dir! with this format's profiles.

name-table clj

(name-table {:keys [entries]})

The synset -> vaelii term map. A type is spelled dog_n and an individual JohannSebastianBach, and a collision — which is what a word with several senses is — takes a numeric suffix in the file-and-offset order the entries arrive in, so the same database always produces the same names.

parse-line clj

(parse-line line)

One data.<pos> record as {:offset :pos :words :pointers :gloss}. :pointers are [symbol target-key] pairs, where a target key is [file offset] — the same key a synset is stored under, so a pointer resolves without knowing which file it crossed into.

nil for a header line, and nil for a record whose fields do not line up — a w_cnt that overruns them, a count that is not a number. with-synsets is what tells those two nils apart and counts the second, so a record this could not read is a drop rather than a synset that quietly never existed.

pointer-readings clj

The pointer symbols this reader has a reading for, and the vaelii predicate each becomes. :genl and :instance are handled structurally rather than as predicates; :inverse names a pointer that is the stored inverse of one already taken.

pos-files clj

The four data files, and the part-of-speech letter each holds. An adjective satellite (s) lives in data.adj and is pointed at as an a, so both letters key the same file — which is what makes a pointer's target resolvable at all.

profiles clj

Named subsets. :taxonomy is the reason most people want WordNet: the genl hierarchy and the instance memberships, without the lexical layer that is nine tenths of the sentences. :nouns keeps that layer but only for nouns.

reader clj

This format's reader, as the seam (vaelii.impl.foreign) hands it out.

translate clj

(translate {:keys [file offset words pointers gloss] :as _synset} names opts)

The sentences one synset becomes: {:context C :sentences [[strength sentence] …] :drops {reason n}}.

Each sentence carries its own strength: the taxonomy is definitional and lands :monotonic, while a gloss, a word form and the lexical relations are observations about usage and land :default.

with-synsets clj

(with-synsets path f)

Call (f synsets) on a lazy seq of every synset in the WordNet directory (or single data file) at path, each carrying its :file. The seq is only valid inside f.

A line that is neither the licence header nor a record parse-line could read travels as {:malformed line} rather than being dropped here: a refusal is counted, not swallowed, and pass 2 is where the count can be kept.

Files are opened one at a time and closed after: the seq is lazy, so the callback is invoked once per file and the results concatenated.