jepsen.nemesis.combined — jepsen 0.1.17

A nemesis which combines common operations on nodes and processes: clock skew, crashes, pauses, and partitions. So far, writing these sorts of nemeses has involved lots of special cases. I expect that the API for specifying these nemeses is going to fluctuate as we figure out how to integrate those special cases appropriately. Consider this API unstable.

This namespace introduces a new abstraction. A nemesis+generator is a map with a nemesis and a generator for that nemesis. This enables us to write an algebra for composing both simultaneously. We call checkers+generators+clients a "workload", but I don't have a good word for this except "nemesis". If you can think of a good word, please let me know.

We also take advantage of the Process and Pause protocols in jepsen.db, which allow us to start, kill, pause, and resume processes.

clock-package clj

(clock-package opts)

A nemesis and generator package for modifying clocks. Options as for nemesis-package.

compose-packages clj

(compose-packages packages)

Takes a collection of nemesis+generators packages and combines them into one. Generators are mixed together randomly; final generators proceed sequentially.

db-generators clj

(db-generators opts)

A map with a :generator and a :final-generator for DB-related operations. Options are from nemesis-package.

db-nemesis clj

(db-nemesis db)

A nemesis which can perform various DB-specific operations on nodes. Takes a database to operate on. This nemesis responds to the following f's:

:start :kill :pause :resume

In all cases, the :value is a node spec, as interpreted by db-nodes.

db-nodes clj

(db-nodes test db node-spec)

Takes a test, a DB, and a node specification. Returns a collection of nodes taken from that test. node-spec may be one of:

nil - Chooses a random, non-empty subset of nodes :one - Chooses a single random node :minority - Chooses a random minority of nodes :majority - Chooses a random majority of nodes :primaries - All nodes which we think are primaries :all - All nodes ["a", ...] - The specified nodes

db-package clj

(db-package opts)

A nemesis and generator package for acting on a single DB. Options are from nemesis-package.

default-interval clj

The default interval, in seconds, between nemesis operations.

grudge clj

(grudge test db part-spec)

Computes a grudge from a partition spec. Spec may be one of:

:one Isolates a single node :majority A clean majority/minority split :majorities-ring Overlapping majorities in a ring :primaries Isolates all primaries into single-node components

nemesis-package clj

(nemesis-package opts)

Takes an option map, and returns a map with a :nemesis, a :generator for its operations, a :final-generator to clean up any failure modes at the end of a test, and a :perf map that can be passed to checker/perf to render nice graphs.

This nemesis is intended for throwing a broad array of simple failures at the wall, and seeing "what sticks". Once you've found a fault, you can restrict the failure modes to specific types of faults, and specific targets for those faults, to try and reproduce it faster.

This nemesis is not intended for complex sequences of faults, like partitionining away a leader, flipping some switch, adjusting the clock on an unrelated node, then crashing someone else. I don't think I can devise a good declarative langauge for that in a way which is simpler than "generators" themselves. For those types of faults, you'll write your own generator instead, but you may be able to use this nemesis to execute some or all of those operations.

Mandatory options:

:db The database you'd like to act on

Optional options:

:interval The interval between operations, in seconds. :faults A collection of enabled faults, e.g. [:partition, :kill, ...] :partition Controls network partitions :kill Controls process kills :pause Controls process pauses and restarts

Possible faults:

:partition :kill :pause :clock

Partition options:

:targets A collection of partition specs, e.g. [:majorities-ring, ...]

Kill and Pause options:

:targets A collection of node specs, e.g. [:one, :all]

nemesis-packages clj

(nemesis-packages opts)

Just like nemesis-package, but returns a collection of packages, rather than the combined package, so you can manipulate it further before composition.

node-specs clj

(node-specs db)

Returns all possible node specification for the given DB. Helpful when you don't know WHAT you want to test.

partition-nemesis clj

(partition-nemesis db)

(partition-nemesis db p)

Wraps a partitioner nemesis with support for partition specs. Uses db to determine primaries.

partition-package clj

(partition-package opts)

A nemesis and generator package for network partitions. Options as for nemesis-package.

partition-specs clj

(partition-specs db)

All possible partition specs for a DB.