The rationale for bitemporality is also explained in this blog post.
A baseline notion of time that is always available is
transaction-time; the point at which data is transacted into the
database.
Bitemporality is the addition of another time-axis: valid-time.
| Time | Purpose |
|---|---|
| Used for audit purposes, technical requirements such as event sourcing. |
| Used for querying data across time, historical analysis. |
transaction-time represents the point at which data arrives into the
database. This gives us an audit trail and we can see what the state
of the database was at a particular point in time. You cannot write a
new transaction with a transaction-time that is in the past.
valid-time is an arbitrary time that can originate from an upstream system, or by default is set to transaction-time. Valid time is what users will typically use for query purposes.
In Crux, when transaction-time isn’t specified, it is set to
now. When writing data, in case there isn’t any specific valid-time
available, valid-time and transaction-time take the same value.
|
In situations where your database is not the ultimate owner of the data—where corrections to data can flow in from various sources and at various times—use of transaction-time is inappropriate for historical queries.
Imagine you have a financial trading system and you want to perform calculations based on the official 'end of day', that occurs each day at 17:00 hours. Does all the data arrive into your database at exactly 17:00? Or does the data arrive in fact arrive from an upstream source, and we have to allow for some data to arrive out of order, and some might just arrive after 17:00?
This can often be the case with high throughput systems where there are clusters of processing nodes, enriching the data before it gets to our store.
In this example, we want our queries to include the straggling bits of data for our calculation purposes, and this is where valid-time comes in. When data arrives into our database, it can come with an arbitrary time-stamp that we can use for querying purposes.
We can tolerate data arriving out of order, as we’re not completely dependent on transaction-time.
For audit reasons, we might wish to know with certainty the value of a
given entity-attribute at a given tx-instant. In this case, we want to
exclude the possibility of the valid past being amended, so we need a
pre-correction view of the data, relying on tx-instant.
To achieve this you can use as-of using ts (valid-time) and tx-ts
(transaction-time).
Recording bitemporal information with your data is essential when dealing with lag, corrections, and efficient auditability:
Lag is found wherever there is risk of non-trivial delay until an event can be recorded. This is common between systems that communicate over unreliable networks.
Corrections are needed as errors are uncovered and as facts are reconciled.
Ad-hoc auditing is an otherwise intensive and slow process requiring significant operational complexity.
With Crux you retain visibility of all historical changes whilst compensating for lag, making corrections, and performing audit queries. By default, deleting data only erases visibility of that data from the current perspective. You may of course still evict data completely as the legal status of information changes.
These capabilities are known to be useful for:
Event Sourcing (e.g. retroactive and scheduled events)
Ingesting out-of-order temporal data from upstream timestamping systems
Maintaining a slowly changing dimension for decision support applications
Recovering from accidental data changes and application errors (e.g. billing systems)
Auditing all data changes and performing data forensics when necessary
Responding to new compliance regulations and audit requirements
Avoiding the need to set up additional databases for historical data and improving end-to-end data governance
Building historical models that factor in all historical data (e.g. insurance calculations)
Accounting and financial calculations (e.g payroll systems)
Development, simulation and testing
Live migrations from legacy systems
Scheduling and previewing future states (e.g. publishing and content management)
Reconciling temporal data across eventually consistent systems
Applied industry-specific examples include:
Legal Documentation – maintain visibility of all critical dates relating to legal documents, including what laws were known to be applicable at the time, and any subsequent laws that may be relevant and applied retrospectively
Insurance Coverage – assess the level of coverage for a beneficiary across the lifecycle of care and legislation changes
Reconstruction of Trades – readily comply with evolving financial regulations
Adverse Events in Healthcare – accurately record a patient’s records over time and mitigate human error
Intelligence Gathering – build an accurate model of currently known information to aid predictions and understanding of motives across time
Criminal Investigations – efficiently organise analysis and evidence whilst enabling a simple retracing of investigative efforts
This example is based on an academic paper.
During a criminal investigation it is critical to be able to refine a temporal understanding of past events as new evidence is brought to light, errors in documentation are accounted for, and speculation is corroborated. The paper referenced above gives the following query example:
The paper then lists a sequence of entry and departure events at various United States border checkpoints. We as the investigator will step through this sequence to monitor a set of suspects. These events will arrive in an undetermined chronological order based on how and when each checkpoint is able to manually relay the information.
Assuming Day 0 for the investigation period is #inst "2018-12-31", the
initial documents are ingested using the Day 0 valid time:
link:./query_test_examples.clj[role=include]The first document shows that Person 2 was recorded entering via :SFO and
the second document shows that Person 3 was recorded entering :LA.
No new recorded events arrive on Day 1 (#inst "2019-01-01"), so there are no
documents available to ingest.
A single event arrives on Day 2 showing Person 4 arriving at :NY:
link:./query_test_examples.clj[role=include]Next, we learn on Day 3 that Person 4 departed from :NY, which is
represented as an update to the existing document using the Day 3 valid time:
link:./query_test_examples.clj[role=include]On Day 4 we begin to receive events relating to the previous days of the investigation.
First we receive an event showing that Person 1 entered :NY on Day 0 which
must ingest using the Day 0 valid time #inst "2018-12-31":
link:./query_test_examples.clj[role=include]We then receive an event showing that Person 1 departed from :NY on Day 3,
so again we ingest this document using the corresponding Day 3 valid time:
link:./query_test_examples.clj[role=include]Finally, we receive two events relating to Day 4, which can be ingested using the current valid time:
link:./query_test_examples.clj[role=include]On Day 5 there is an event showing that Person 2, having arrived on Day 0
(which we already knew), departed from :SFO on Day 5.
link:./query_test_examples.clj[role=include]No new recorded events arrive on Day 6 (#inst "2019-01-06"), so there are no
documents available to ingest.
On Day 7 two documents arrive. The first document corrects the previous
assertion that Person 3 departed on Day 4, which was misrecorded due to human
error. The second document shows that Person 3 has only just departed on Day
7, which is how the previous error was noticed.
link:./query_test_examples.clj[role=include]Two documents have been received relating to new arrivals on Day 8. Note that
Person 3 has arrived back in the country again.
link:./query_test_examples.clj[role=include]On Day 9 we learn that Person 3 also departed on Day 8.
link:./query_test_examples.clj[role=include]A single document arrives showing that Person 5 entered at :LA earlier that
day.
link:./query_test_examples.clj[role=include]Similarly to the previous day, a single document arrives showing that Person
7 entered at :NY earlier that day.
link:./query_test_examples.clj[role=include]Finally, on Day 12 we learn that Person 6 entered at :NY that same day.
link:./query_test_examples.clj[role=include]Let’s review the question we need to answer to aid our investigations:
We are able to easily express this as a query in Crux:
link:./query_test_examples.clj[role=include]The answer given by Crux is a simple set of the three relevant people along with the details of their last entry and confirmation that none of them were known to have yet departed at this point:
link:./query_test_examples.clj[role=include]Can you improve this documentation?Edit on GitHub
cljdoc is a website building & hosting documentation for Clojure/Script libraries
× close