jepsen.core — jepsen 0.1.14

Entry point for all Jepsen tests. Coordinates the setup of servers, running tests, creating and resolving failures, and interpreting results.

Jepsen tests a system by running a set of singlethreaded processes, each representing a single client in the system, and a special nemesis process, which induces failures across the cluster. Processes choose operations to perform based on a generator. Each process uses a client to apply the operation to the distributed system, and records the invocation and completion of that operation in the history for the test. When the test is complete, a checker analyzes the history to see if it made sense.

Jepsen automates the setup and teardown of the environment and distributed system by using an OS and client respectively. See run! for details.

analyze! clj

(analyze! test)

After running the test and obtaining a history, we perform some post-processing on the history, run the checker, and write the test to disk again.

client-worker clj

(client-worker test process-id node)

A worker for executing operations on clients. Takes a test, an initial process id, and a node to bind clients to.

conj-op! clj

(conj-op! test op)

Add an operation to a tests's history, and returns the operation.

invoke-op! clj

(invoke-op! op test client abort?)

Applies an operation to a client, catching client exceptions and converting them to infos. Returns a completion op, throwing if the completion is invalid.

log-results clj

(log-results test)

Logs info about the results of a test to stdout, and returns test.

maybe-snarf-logs! clj

(maybe-snarf-logs! test)

Snarfs logs, swallows and logs all throwables. Why? Because we do this when we encounter an error and abort, and we don't want an error here to supercede the root cause that made us abort.

nemesis-apply-op! clj

(nemesis-apply-op! op test nemesis abort?)

Logs, journals, and invokes an operation, logging and journaling its completion, and returning the completed operation.

nemesis-invoke-op! clj

(nemesis-invoke-op! op test client abort?)

Applies an operation to a nemesis, catching exceptions and converting them to infos. Returns a completion op, throwing if the completion is invalid.

nemesis-worker clj

(nemesis-worker test)

A worker for introducing failures. Takes a test.

primary clj

(primary test)

Given a test, returns the primary node.

real-pmap clj

(real-pmap f coll)

Real-pmap throws the first exception it gets, which might be something unhelpful like InterruptedException or BrokenBarrierException. This variant works like real-pmap, but throws more interesting exceptions when possible.

run! clj

(run! test)

Runs a test. Tests are maps containing

:nodes A sequence of string node names involved in the test :concurrency (optional) How many processes to run concurrently :ssh SSH credential information: a map containing... :username The username to connect with (root) :password The password to use :port SSH listening port (22) :private-key-path A path to an SSH identity file (~/.ssh/id_rsa) :strict-host-key-checking Whether or not to verify host keys :logging Logging options; see jepsen.store/start-logging! :os The operating system; given by the OS protocol :db The database to configure: given by the DB protocol :client A client for the database :nemesis A client for failures :generator A generator of operations to apply to the DB :checker Verifies that the history is valid :log-files A list of paths to logfiles/dirs which should be captured at the end of the test. :nonserializable-keys A collection of top-level keys in the test which shouldn't be serialized to disk.

Tests proceed like so:

1. Setup the operating system

2. Try to teardown, then setup the database

• If the DB supports the Primary protocol, also perform the Primary setup on the first node.

3. Create the nemesis

4. Fork the client into one client for each node

5. Fork a thread for each client, each of which requests operations from the generator until the generator returns nil

• Each operation is appended to the operation history
• The client executes the operation and returns a vector of history elements
  • which are appended to the operation history

6. Capture log files

7. Teardown the database

8. Teardown the operating system

9. When the generator is finished, invoke the checker with the history

• This generates the final report

run-case! clj

(run-case! test)

Spawns nemesis and clients, runs a single test case, and returns that case's history.

run-workers! clj

(run-workers! workers)

Runs a set of workers through setup, running, and teardown.

snarf-logs! clj

(snarf-logs! test)

Downloads logs for a test. Updates symlinks.

synchronize clj

(synchronize test)

(synchronize test timeout-s)

A synchronization primitive for tests. When invoked, blocks until all nodes have arrived at the same point.

This is often used in IO-heavy DB setup code to ensure all nodes have completed some phase of execution before moving on to the next. However, if an exception is thrown by one of those threads, the call to synchronize will deadlock! To avoid this, we include a default timeout of 60 seconds, which can be overridden by passing an alternate timeout in seconds.

uninteresting-exceptions clj

Exceptions which are less interesting to us.

with-db clj macro

(with-db test & body)

Wraps body in DB setup and teardown.

with-log-snarfing clj macro

(with-log-snarfing test & body)

Evaluates body and ensures logs are snarfed afterwards. Will also download logs in the event of JVM shutdown, so you can ctrl-c a test and get something useful.

with-os clj macro

(with-os test & body)

Wraps body in OS setup and teardown.

with-resources clj macro

(with-resources [sym start stop resources] & body)

Takes a four-part binding vector: a symbol to bind resources to, a function to start a resource, a function to stop a resource, and a sequence of resources. Then takes a body. Starts resources in parallel, evaluates body, and ensures all resources are correctly closed in the event of an error.

Worker clj protocol

Polymorphic lifecycle for worker threads; synchronized setup, run, and teardown phases, each with error recovery. Workers are singlethreaded and may be stateful. Return value are ignored.

abort-worker! clj

(abort-worker! worker)

run-worker! clj

(run-worker! worker)

setup-worker! clj

(setup-worker! worker)

teardown-worker! clj

(teardown-worker! worker)

worker-name clj

(worker-name worker)