lite.target.compose — com.igel-data/jepsen-lite 0.1.0

The :compose target-type: the target runs as docker-compose services that Lite brings up and takes down, with faults injected into the containers by Pumba.

This is the only target-type that can partition anything, and the reason is the same one that runs through the whole axis: a fault is possible when Lite owns the thing the fault happens to. :in-process owns an object, so it can destroy it. :local-process owns a process, so the kernel will kill or stop it. :compose owns a container -- and a container has a network interface of its own, which is what makes cutting it off meaningful. Lite cannot partition a process that reaches it over loopback, and it never pretends otherwise.

Faults are Pumba's, not Lite's:

crash pumba kill --signal SIGKILL, then the service is brought back up pause pumba pause --duration d (freezes the container's processes) partition pumba netem loss 100% (the container's traffic vanishes)

pause and partition heal themselves when their duration expires, so each is a single op rather than a start/stop pair. crash doesn't: something has to start the container again, and that something is Lite.

Config:

{:type :compose :file "examples/compose/docker-compose.yml" :service "kvs" ; which service to run and hit :container "jepsen-lite-kvs" ; its container_name, for Pumba :url "http://127.0.0.1:8080" ; its published endpoint :ready-timeout 120000 ; ms to wait for it to listen :fault-duration "5s" ; how long pause/partition last :pumba-image "ghcr.io/alexei-led/pumba" :tc-image "ghcr.io/alexei-led/pumba-alpine-nettools:latest"}

Pumba runs as a container itself, with the docker socket mounted, so there is nothing to install beyond Docker. netem needs tc, which the target's own image is unlikely to have; --tc-image lends it one, which is why it is configured rather than assumed.

compose-command clj

(compose-command {:keys [file]} & args)

docker compose -f <file> <args...>, run in the compose file's project.

crash! clj

(crash! {:keys [adapter config conn crashes lifecycle-lock]})

SIGKILL inside the container, then bring the service back.

Pumba does the killing; the restart is Lite's, because a container that is gone stays gone unless something starts it -- a restart: policy in the compose file would race with this, so up -d is issued either way and is a no-op if Docker got there first. As with :local-process, the new connection is published before the old one is closed, so ops in the window meet a dead target rather than a nil one.

crash-count clj

(crash-count target)

down-command clj

(down-command config)

down -v: the volumes go too, so the next run starts from an empty store rather than inheriting the last one's data.

kill-command clj

(kill-command {:keys [container] :as config})

partition! clj

(partition! {:keys [config]})

Drops everything the container sends, for the fault's duration. Requests still arrive, so an op in the window may well take effect with nobody left to tell -- which is what :info means.

partition-command clj

(partition-command {:keys [container fault-duration tc-image] :as config})

100% egress loss for the duration: the container is still running and still accepting requests, and nothing it says gets out. Ops in the window time out, which is indeterminate and not failure -- the target may well have applied them.

pause! clj

(pause! {:keys [config]})

Freezes every process in the container for the fault's duration, after which Pumba unfreezes it. Returns the duration in seconds -- a number, because checkers read every op in the history, nemesis ops included.

pause-command clj

(pause-command {:keys [container fault-duration] :as config})

pumba-command clj

(pumba-command {:keys [pumba-image]} & args)

Pumba, as a container with the docker socket mounted, so nothing has to be installed for it. args is the fault and its options.

up-command clj

(up-command {:keys [service] :as config})

--wait, because a published port answers before the service does: Docker binds the host side of the mapping as soon as the container is created, so a connection succeeds -- and is closed again immediately -- while the process inside is still starting. Ops issued into that window fail for a reason that has nothing to do with the target.

--wait hands the question to the service's own healthcheck:, which is where it belongs: only the service knows what ready means for it. Without one, Docker waits for the container to be running and no more, and Lite's endpoint probe is all that stands between the run and that window.