Datalog Queries

Extensible Data Notation (edn) is a simple data format that is used to describe XTDB Datalog queries. To understand how edn works, read the brief description at http://edn-format.org.

To learn about Datalog queries and get familiar with the concepts, read the Learn XTDB Datalog Today tutorial.

A query in XTDB is performed by calling xt/q on an XTDB database snapshot with a quoted map and, optionally, additional arguments.

The query map accepts the following Keywords

You can directly specify a logic variable from your query. The following will return all last names bound to the logic variable n.

You can use basic Clojure-like expressions within a :find parameter.

You can specify an aggregate function to apply to at most one logic variable.

Custom (user-defined) aggregates are supported by adding a new method (via Clojure defmethod) for xtdb.query/aggregate. This method takes a single (ignored) parameter and returns a multi-arity function which accepts zero parameters, an accumulator, or an accumulator and a single entity. For example:

XTDB queries support a pull syntax, allowing you to decouple specifying which entities you want from what data you’d like about those entities in your queries. XTDB’s support is based on the excellent EDN Query Language (EQL) library.

To specify what data you’d like about each entity, include a (pull ?logic-var projection-spec) entry in the :find clause of your query:

You can quickly grab the whole document by specifying * in the projection spec:

If you have the entity id(s) in hand, you can call pull or pull-many directly:

We can navigate to other entities (and hence build up nested results) using joins. Joins are specified in {} braces in the projection-spec - each one maps one join key to its nested spec:

We can also navigate in the reverse direction, looking for entities that refer to this one, by prepending _ to the attribute name:

To return maps rather than tuples, supply the map keys under :keys for keywords, :syms for symbols, or :strs for strings:

Clauses may refer to combinations of literal values and logic variables as inputs. Identical logic variables used across multiple clauses unify automatically (with the exception of or-join or not-join scoping).

A literal set containing literal values (e.g. #{"val-1" "val-2"}) will be interpreted as an input relation of distinct values, and can be used in place of any literal input (e.g. a set of entity IDs in the first position of a triple clause).

A triple clause is a vector of (1) a literal entity ID or a logic variable, (2) a hard-coded attribute keyword (top-level key in a document), and (3) optionally, a value which can be a literal or a logic variable.

It restricts results by matching EAV facts

Note that the keyword attribute must always be specified and therefore you cannot use a logic variable in the attribute position. You can however enumerate combinations of known attributes using rules, or execute independent queries efficiently using openDB (open-db). A list of all known attributes is available via the attributeStats (attribute-stats) API.

Any fully qualified Clojure function that returns a boolean can be used as a "filter" predicate clause.

Predicate clauses must be placed in a clause, i.e. with a surrounding vector.

This matches all entities, p which have an odd :age.

You can nest a subquery with a :where clause to bind the result for further use in the query.

A range predicate is a vector containing a list of a range operator and then two logic variables or literals.

Allowed range operators are <, <=, >=, >, and =.

Use a unification predicate, either == or !=, to constrain two independent logic variables. Literals (and sets of literals) can also be used in place of one of the logic variables.

The not clause rejects a graph if all the clauses within it are true.

This will match any document which does not have a :name of "Ivan" and a :last-name of "Ivanov".

The not-join rule allows you to restrict the possibilities for logic variables by asserting that there does not exist a match for a given sequence of clauses.

You declare which logic variables from outside the not-join scope are to be used in the join.

Any other logic variables within the not-join are scoped only for the join.

This will match any entity, p, which has different values for the :name and :last-name field.

Importantly, the logic variable n is unbound outside the not-join clause.

An or clause is satisfied if any of its legs are satisfied.

This will match any document, p, which has a :last-name of "Ivanov" or "Ivannotov".

When within an or rule, you can use and to group clauses into a single leg (which must all be true).

Whenever the query engine complains that each leg in the or or or-join clause requires the "same logic variables", you can add a no-op predicate clause like [(any? e)] within and clauses for each of the missing variables in the various legs. You need to add such no-op predicates until each leg contains the same set of variables.

Hypothetically, XTDB could automatically detect these cases and insert no-op predicates on the user’s behalf, but that would be a deviation from the essential semantics of Datalog. Note that clojure.core/any? is a function that always returns true.

The or-join clause is satisfied if any of its legs are satisfied.

You declare which logic variables from outside the or-join scope are to be used in the join.

Any other logic variables within the or-join are scoped only for the join.

This will match any document, p which has an :age greater than or equal to 18 or has a :name of "Ivan".

Importantly, the logic variable a is unbound outside the or-join clauses.

See Rules

In the above query, we parameterize the first-name symbol, and pass in "Ivan" as our input, binding "Ivan" to first-name in the query.

Result Set:

This query shows binding to a collection of inputs - in this case, binding first-name to all of the different values in a collection of first-names.

Result Set:

In this query we are binding a set of variables to a single value each, passing in a collection as our input. In this case, we are passing a collection with a first-name followed by a last-name.

Result Set:

Here we see how we can extend the parameterisation to match using multiple fields at once by passing and destructuring a relation containing multiple tuples.

Result Set:

To improve the performance of a rule you can specify that certain arguments in the rule head must be "bound" logic variables (i.e. there must be known values for each argument at the point of evaluation) by enclosing them in a vector in the first argument position. Any remaining arguments will be treated as regular "free" logic variables.

As an analogy, bound variables are input arguments to a function, and free variables are the destructured return values from that function.

Changes are only necessary in the rule head(s) - no changes are required in the body or the usage clauses. Rule heads must always match.

For example, the following query and rule set will work and return the correct results.

However, specifying that the p variable should be bound before the rule can be evaluated will improve the evalution time by many orders-of-magnitude for large data sets.

Given the following documents in the database

We can run a query to return a set of tuples that satisfy the join on the attribute :name

Result Set:

Note that every person joins once, plus 2 more matches.

Given the following documents in the database

We can run a query to return a set of entities that :follows the set of entities with the :name value of "Ivan"

Result Set:

Note that because XTDB is schemaless there is no need to have elsewhere declared that the :follows attribute may take a value of edn type set.

XTDB allows you to retrieve all versions of a given entity:

When retrieving the previous versions of an entity, you have the option to additionally return the documents associated with those versions (by using :with-docs? in the additional options map)

Retrievable entity versions can be bounded by four time coordinates:

All coordinates are inclusive. All coordinates can be null.

See the open-entity-history (Clojure) and openEntityHistory (Java) APIs.

Logic variables used in queries must always be quoted in the :find and :where clauses, which in the most minimal case could look like the following:

However it is often convenient to quote entire clauses or even the entire query map rather than each individual use of every logic variable, for instance:

Note that use of Clojure’s syntax quoting may cause confusion and is therefore not recommended. This is because Datalog built-ins like range predicates, unification predicates and rules (or, not etc.) will be mistakenly coerced into corresponding Clojure core functions (clojure.core/or, clojure.core/== etc.) and produce invalid queries.

Say you have a document like so and you want to add it to an XTDB db:

XTDB breaks down vectors into individual components so the query engine is able see all elements on the base level. As a result of this the query engine is not required to traverse any structures or any other types of search algorithm which would slow the query down. The same thing should apply for maps so instead of doing :pocket {:left thing :right thing} you should put them under a namespace, instead structuring the data as :pocket/left thing :pocket/right thing to put the data all on the base level. Like so:

To query inside these vectors the code would be:

Note that l and p is returned as a single element as XTDB decomposes the vector