vaelii.foreign.atomic — com.vaelii/vaelii-foreign 0.1.0

Read ATOMIC-2020 — a commonsense knowledge graph of if-then inferences — and translate it into a vaelii corpus.

Of everything this repo reads, ATOMIC is the one whose content is already the shape vaelii's assumption strengths were built for. Cyc marks a handful of assertions :default; an OBO ontology and an OWL vocabulary are definitional throughout. Every one of ATOMIC's 1.3 million tuples is a defeasible generalization — if PersonX abandons the cat, PersonX probably wants to find a new home for it — collected by asking people what usually follows. Nothing in it is a definition, so nothing in it is written :monotonic, and a corpus where every sentence can be retracted by a later one is a corpus a truth-maintenance system has something to do with.

The format

ATOMIC-2020 ships as tab-separated triples, split into train.tsv, dev.tsv and test.tsv:

PersonX abandons ___ altogether \t xIntent \t to be selfish
PersonX abandons ___ altogether \t xEffect \t none
baseball bat \t ObjectUse \t hit a ball

The 23 relations fall into three families, and the families are this corpus's contexts:

• social — what a person intends, needs, feels, wants, or becomes because of an event (xIntent, xNeed, xAttr, xEffect, xReact, xWant, xReason, and the o* versions for the other people in the event);
• event — how events sit next to each other (isAfter, isBefore, Causes, HasSubEvent, HinderedBy, isFilledBy);
• physical — what a thing is for, where it is, what it is made of (ObjectUse, AtLocation, MadeUpOf, HasProperty, CapableOf, Desires, NotDesires).

Nodes are phrases, and stay phrases

An ATOMIC node is a fragment of English — PersonX abandons ___ altogether, to be selfish, baseball bat. It is not a term and has no definition anywhere, so each one becomes an individual named from its own text, with the text itself kept alongside as (nodeText PersonXAbandonsAltogether "PersonX abandons ___ altogether").

The temptation is to make baseball bat a type and PersonX abandons ___ altogether an event template, and it is worth saying why that is not done: ATOMIC never marks which of its nodes are kinds, the same string appears on both sides of relations from different families, and a type with no members and no supertype is a name with nothing attached. Inventing a type system the corpus does not have would make the output look more structured than the source and be wrong in ways nothing could check.

Two rows with the same text are the same node, which is what makes the result a graph rather than a list: to be selfish reached as PersonX's intent and reached again as somebody's reaction is one individual with two relations to it.

A none tail — ATOMIC's own marker for "annotators said nothing follows" — is dropped and counted. It is a real annotation and it is not a fact.

classify clj

(classify rows)

(classify rows opts)

Pass 1. {:nodes #{text …} :relations #{name …} :rows n}.

Every node is an individual, so there is no role to resolve — what pass 1 is for here is the set of them: a node reached by two rows must be one term, and that cannot be known one row at a time.

:empty-tails? has to be read here as well as in translate: an empty tail kept by the flag needs a name, and pass 2 has no way to mint one.

convert! clj

(convert! path out-dir)

(convert! path out-dir opts)

Convert the ATOMIC release at path — a directory of .tsv files, or one file — into a vaelii corpus under out-dir. Two passes: the first collects the nodes and names them, the second writes the sentences. Returns the report map.

Options: :node-text? writes a nodeText fact carrying each node's original English (on — it is the only way back to the source); :empty-tails? keeps the rows whose tail is the none annotation (off — see drop-flags); :limit reads only the first n rows.

drop-flags clj

The convert option that keeps each :filtered drop — see cyc/drop-flags for the contract and plugin-test for what enforces it.

--empty-tails is a flag and not a justification because the objection to importing these is a judgement rather than a fact about the format: writing them states, of a great many events, that they cause a thing called None, which is a bad reading of "the annotators looked and found nothing". Whether that is worse than losing the annotation entirely depends on what the corpus is for, and this is not the place to decide it.

drop-kinds clj

What each of this reader's drop reasons is — see corpus/drop-kinds. A reason missing here counts as :unread.

:no-tail is ATOMIC's none — the annotators' own marker for "nothing follows". It is a real annotation and it is not a fact, so declining to write it is a policy and not a failure to read one.

load-dir! clj

(load-dir! kb dir)

(load-dir! kb dir opts)

Load the corpus at dir into kb — vaelii.foreign.corpus/load-dir! with this format's profiles.

name-table clj

(name-table {:keys [nodes relations]})

The node text -> vaelii term map, plus a predicate per relation. Sorted by text so a collision — two long phrases that abbreviate to the same head — resolves the same way on every run.

parse-line clj

(parse-line line)

One TSV row as {:head :relation :tail}, or nil for a blank line, a header row, or a row without three fields. Surrounding whitespace goes; internal whitespace and the ___ blank are content.

profiles clj

Named subsets, one per relation family — which is how ATOMIC is actually used, since the social half and the physical half answer different questions. :ontology keeps every relation and drops the node texts, which are a third of the sentences.

reader clj

This format's reader, as the seam (vaelii.impl.foreign) hands it out.

relations clj

The 23 ATOMIC-2020 relations, and the family — this corpus's context — each belongs to. A relation outside this table is read as an ordinary predicate in the event context and counted, so a release that adds one is imported rather than silently dropped.

translate clj

(translate row names)

(translate {:keys [head relation tail]} names opts)

The sentences one row becomes, or {:dropped reason}. Everything lands :default: an ATOMIC tuple is what people said usually follows, and there is no other strength that is honest about it.

with-rows clj

(with-rows path f)

Call (f rows) on a seq of every row in the ATOMIC release at path — a directory of .tsv files, or one file. .gz is decompressed on the way through. The seq is only valid inside f.