Datalog Transactions

Transactions are how you insert and modify data within XTDB.

Transactions are atomic, and comprised of a sequence of operations to be performed.

If the transaction contains pre-conditions, all pre-conditions must pass, or the entire transaction is aborted. This processing happens at each node during indexing, and not when submitting the transaction.

Java

A transaction is performed by calling .submitTx on an IXtdbSubmitAPI with a Transaction object.

The Transaction object can be created by either using Transaction.Builder directly or using a Consumer.

If using the Consumer approach, we recommend importing xtdb.api.tx.Transaction.buildTx statically for brevity.

link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

// To run the transaction:
link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

// To run a transaction directly:
link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

Clojure

A transaction is performed by calling xtdb.api/submit-tx on a node with a list of transaction operations.

(xt/submit-tx node
                [
                ;; Operations
                ])

Operations

There are five transaction (write) operations:

Table 1. Write Operations
Operation	Purpose	Pre-condition?
Put	Write a version of a document
Delete	Deletes a specific document
Match	Check the document state against the given document	✓
Evict	Evicts a document entirely, including all historical versions
Function	Runs a transaction function	✓

Java: You can add individual operations to the Transaction.Builder instance with their respective methods.
Clojure: Transaction Operations are vectors which have their associated keyword as their first value.

Put

Puts a Document into XTDB. If a document already exists with the same id, a new version of this document will be created at the supplied valid time. See Valid Time for more details.

Java

link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

1	Putting a document as of now.
2	Putting a document with a specific valid time
3	Putting a document with a valid time and end valid time

Clojure

link:example$test/xtdb/docs/examples/transactions/transactions_test.clj[role=include]

1	Document to add
2	(optional) `valid time`
3	(optional) `end valid time`

Delete

Deletes a Document . See Valid Time for details on how this interacts with multiple versions of the document.

Java

link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

1	Deletes as of now
2	Deleting with a specific valid time
3	Deleting with a valid time and end valid time

Clojure

link:example$test/xtdb/docs/examples/transactions/transactions_test.clj[role=include]

1	Document ID to be deleted
2	(optional) `valid time`
3	(optional) `end valid time`

Match

Match checks the state of an entity - if the entity doesn’t match the provided document, the transaction will not continue.

Use the hasTxCommitted (tx-committed?) API to check whether the transaction was successfully committed or not due to a failed match operation.

Java

link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

1	Passes if `document1` is exactly present now
2	Passes if `document2` is exactly present at `validTime1`
3	Passes if there is no document with the id `documentId1` present now
4	Passes if there is no document with the id `documentId2` present at `validTime2`
5	Operation(s) to apply if all preconditions are met

Clojure

link:example$test/xtdb/docs/examples/transactions/transactions_test.clj[role=include]

1	ID to be matched (for an entity which may or may not exist)
2	A specific document revision (or `nil`)
3	(optional) `valid time`
4	Operation(s) to perform if the document is matched

If the document supplied is nil, the match only passes if there does not exist a document with the given ID.

Evict

Evicts a document from XTDB. Historical versions of the document will no longer be available.

Java

link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

Clojure

link:example$test/xtdb/docs/examples/transactions/transactions_test.clj[role=include]

Evict is primarily used for GDPR Right to Erasure compliance.

It is important to note that Evict is the only operation which will have effects on the results returned when querying against an earlier Transaction Time.

Transaction Functions

Transaction functions are user-supplied functions, defined using Clojure, that will run on each individual node where a transaction is being ingested and processed after having been read from the transaction log. They are conceptually similar to "stored procedures" in traditional databases, except they can only return new transaction operations.

Transaction functions can be used, for example, to safely check the current database state before applying a transaction, for integrity checks, or to patch an entity.

Anatomy

link:example$test/xtdb/docs/examples/transactions/transactions_test.clj[role=include]

1	Transaction functions are passed a `context` parameter and any number of other parameters.
2	The `context` parameter can be used to obtain a database value using `db` or `open-db`.
3	Transaction functions should return a list of transaction operations or `false`

If a list of transaction operations is returned, these are indexed as part of the transaction.

If false is returned, or an exception is thrown, the whole transaction will roll back.

The context reflects the speculative accumulation of all transaction operations applied up to the current point in the processing.

Creating / Updating

Transaction functions are created/updated by submitting a document to XTDB with the desired function.

Java

You create a function document with XtdbDocument.createFunction.

It takes the ID for the function as well as a string consisting of the Clojure function to run.

link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

Clojure

The document should use the :xt/fn key (note, not ::xt/fn).

link:example$test/xtdb/docs/examples/transactions/transactions_test.clj[role=include]

1	Note that the function itself is quoted

Usage

When invoking a transaction function, you specify its ID and (optionally) other arguments

Java

link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

Clojure

Note that the transaction function operation ::xt/fn is Clojure shorthand for :xtdb.api/fn (assuming the xt require alias is used) - be careful not to confuse this keyword with the :xt/fn keyword used inside the transaction function document.

link:example$test/xtdb/docs/examples/transactions/transactions_test.clj[role=include]

1	Function ID
2	Parameter(s)

Transaction functions may return further transaction function invocation operations, which will in turn expand recursively until there are only primitive transaction operations remaining for the given transaction.

Documents

Java

A XtdbDocument is created with an ID that must be of a valid type.

The instance itself is immutable and plusing/minusing data yields a new instance of XtdbDocument.

Similarly to Transactions, you can use XtdbDocument.Builder directly or use the Consumer approach.

link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

// You can also chain creating new instances of the document,
// but this will be slow for larger documents.
link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

Clojure

A document is a map from keywords to values.

{:xt/id :dbpedia.resource/Pablo-Picasso
 :name "Pablo"
 :last-name "Picasso"}

All documents must contain the :xt/id key.

Persistence of Clojure metadata is not supported (although is not currently rejected either) - we strongly recommended that all metadata should be removed from each document prior to submission to XTDB, to avoid potential equality checking issues (see here for more details).

For operations containing documents, the id and the document are hashed, and the operation and hash is submitted to the tx-topic in the event log. The document itself is submitted to the doc-topic, using its content hash as key. In Kafka, the doc-topic is compacted, which enables later eviction of documents.

Valid IDs

Java

The following types of document IDs are allowed:

Table 2. Valid ID types (Java)
Type	Example
Keyword	`Keyword.intern("my-id")`
String	`"my-id"`
Long	`42L`
UUID	`UUID.randomUUID()`
URI	`URI.create("mailto:hello@xtdb.com")`
IPersistentMap	`PersistentArrayMap.EMPTY.assoc("foo", "bar")`

URLs are valid IDs for historical reasons, but discouraged due to their hashCode depending on a DNS lookup.

Clojure

The following types of :xt/id are allowed:

Table 3. Valid ID types (Clojure)
Type	Example
Keyword	`:my-id` or `:dbpedia.resource/Pablo-Picasso`
String	`"my-id"`
Integers/Longs	`42`
UUID	`#uuid "6f0232d0-f3f9-4020-a75f-17b067f41203"`
URI	`(URI. "mailto:hello@xtdb.com")`
Maps	`{:this :id-field}`

URLs are valid IDs for historical reasons, but discouraged due to their hashCode depending on a DNS lookup.

Indexing

XTDB automatically indexes the top-level fields across all documents as Entity-Attribute-Value triples. Vector and Set values are indexed as cardinality-many values, i.e. producing multiple EAVs.

Transaction Time

When you submit a transaction, the current time will be the Transaction Time.

You can override the transaction time for a transaction (e.g. for importing data into XTDB from another bitemporal database) so long as:

the transaction time provided is no earlier than any other transaction currently in the system - i.e. transaction times must be increasing.
the transaction time is no later than the clock on the transaction log (e.g. Kafka) - i.e. transactions cannot be inserted into the future.

Java

buildTx(tx -> {
  tx.put(document);
  tx.withTxTime(txTime);
});

Clojure

(xt/submit-tx node
              [[::xt/put {:xt/id :foo}]]
              {::xt/tx-time #inst "2020"})

Valid Times

When an optional valid time is omitted from a transaction operation, the Transaction Time will be used as valid time.

Only Valid Time Provided

When valid time is used as a parameter for Put or Delete, it specifies the time from which the change is valid.

If there is a document, A, already present at the given valid time, the operation’s effect is valid from valid time to the end of the current validity of A.

End Valid Time

When both valid time and end valid time are provided for Put or Delete, the operation’s effect is valid from valid time (inclusive) to end valid time (exclusive).

Awaiting Transactions

After a transaction is submitted, it needs to be indexed before it is visible in XTDB DB snapshots.

The return value from submitting the transaction can be used to wait for the transaction to be indexed.

This return value holds both the ID of the transaction, and the Transaction Time

Java

In Java, you receive a TransactionInstant from the submitTx call.

link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

Clojure

In Clojure, you receive a map from submit-tx containing ::xt/tx-id and ::xt/tx-time

link:example$test/xtdb/docs/examples/transactions/transactions_test.clj[role=include]

Speculative transactions

You can submit speculative transactions to XTDB, to see what the results of your queries would be if a new transaction were to be applied. This is particularly useful for forecasting/projections or further integrity checks, without persisting the changes or affecting other users of the database.

You’ll receive a new database value, against which you can make queries and entity requests as you would any normal database value. Only you will see the effect of these transactions - they’re not submitted to the cluster, and they’re not visible to any other database value in your application.

Java

You submit these transactions to an instance of IXtdbDatasource using withTx:

link:example$test/xtdb/docs/examples/transactions/TransactionsTest.java[role=include]

Clojure

You submit these transactions to a database value using with-tx:

link:example$test/xtdb/docs/examples/transactions/transactions_test.clj[role=include]
link:example$test/xtdb/docs/examples/transactions/transactions_test.clj[role=include]
link:example$test/xtdb/docs/examples/transactions/transactions_test.clj[role=include]
link:example$test/xtdb/docs/examples/transactions/transactions_test.clj[role=include]

The entities submitted by the speculative Put take their valid time (if not explicitly specified) from the valid time of the db from which they were forked.

Dropping the database

XTDB does not currently provide any API to perform a "drop" operation (physical deletion) of the entire database. To drop the database, you must first manually call the close method on the node(s), and then delete both the index-store directory and the relevant data (or directories) for the backend modules used for the transaction log and document store components (as per your configuration).

For testing purposes you can use Files/createTempDirectory or equivalent.

❮Datalog Queries Language Reference❯