How dsfuzz/mallory works
Main Components
• Jepsen Test Harness (Control Node):
– Written in Clojure, it contains the “jepsen” directory and a suite of tests (e.g., tests/jepsen/* for various systems such as braft, dqlite, mongodb, redisraft, scylladb, tikv, and even mallory adaptations).
– It executes test flows by spawning “processes” with clients, generators, checkers, and the nemesis.
– Show the flow: the test harness “controls” when clients perform operations and when faults are scheduled.
• Mallory (Adaptive Nemesis):
– Implemented in Rust (see directories such as mediator, cov-server, and instrumentor, which are built with Rust/Cargo).
– Acts as an enhanced nemesis that hooks into a Jepsen test and injects faults based on runtime observation.
– Clearly illustrate its integration with the Jepsen control node and its interaction with the system under test (SUT).
• Distributed System Under Test (SUT):
– Show that the fault injection (via Mallory) disrupts or monitors these nodes.
• Auxiliary Services:
– Cov-server: Likely a Rust web service (using Rocket, as inferred from Rocket.toml) that might be responsible for handling coverage or other server-side functionalities.
– Mediator: Acts as an intermediary, perhaps aggregating feedback or coordinating test events among various components (its own README and Rocket configuration suggest it is a networking server).
– Instrumentor: Contains LLVM mode components suggesting that it is used to instrument or gather runtime data from the system under test.
• Docker and Environment Setup:
– The “docker” directory, along with configuration files (docker-compose, Vagrant, setup scripts, and control templates), represents the containerized environment in which the tests run.
– It provides the orchestration layer to spin up control nodes, test nodes, and any additional services required.
Relationships and Interactions
• Jepsen test harness (control node) issues SSH commands to launch and interact with “db nodes.”
• Docker orchestration underpins the entire environment; include a layer that encapsulates various containerized components (control node, test nodes, mediator, instrumentor, etc.).
Architectural Patterns and Design Principles
• Modularity: Separate components for testing (Jepsen), fault injection (Mallory), instrumenting (instrumentor), and supporting services (mediator, cov-server).
• Client/Server Model: The Jepsen control node works as the server that schedules operations on multiple client nodes and communicates with Mallory.
• Containerization: Emphasize the role of Docker (and Vagrant for development) in encapsulating service environments.
Including Technologies and Frameworks
• Clojure for the Jepsen test harness; note that it leverages libraries for SSH, history collection, and checkers.
• Rust for Mallory (and other components such as cov-server and mediator), with Cargo as the build system.
• Rocket for configuring/hosting the Rust web server components.
• Docker and Docker Compose for orchestration; include Vagrant for environment set-up.
• LLVM/instrumentation aspects via the instrumentor directory.