How rub-syssec/nyx works
Project type and purpose
• Nyx is an open-source greybox hypervisor fuzzer for type-2 hypervisors (bhyve, QEMU).
• It builds on three main components:
• KVM-PT: a Linux kernel module that exposes Intel-PT tracing per vCPU, nested hypercalls, dirty-memory tracking, etc.
• QEMU-PT: a patched QEMU (user-space) that implements corresponding hypercall handlers, fast snapshots, and an Intel-PT decoder.
• Nyx-Fuzzer: a Rust-based fuzzing engine (cargo “rust_fuzzer” crate + fuzz_runner + helpers + hypervisor_spec) orchestrating the fuzzing loop, snapshot management, and corpus.
File structure
• Top‐level directories:
• KVM-PT/ – the standalone LKM source + build scripts (load.sh) + headers.
• QEMU-PT/ – QEMU patch + compile script + QEMU source archive.
• Targets/ – per-hypervisor fuzz targets (bhyve & QEMU) each containing VM setup, “agent” code (guest-side loader, req_data, set_ip_range), config templates.
• nyx_fuzzer/ – the Rust workspace:
– config/ – config crate for shared spec formats (msgpack)
– fuzz_runner/ – binary crate implementing forkserver, coverage reporting loop, affine-type harness
– helpers/ – common Rust utilities (hash_by_ref, etc.)
– hypervisor_spec/ – Python- and C-based tooling to generate custom micro-OS images (“hypertrash_os”), ISO, GRUB patches, Tesseract intermediate decompiler, etc.
– rust_fuzzer/ – the main fuzzer binary (“cargo run ‑-release”)
– structured_fuzzer/ – a separate Rust crate for generative, structured-input fuzzing (examples, interpreter, spec_lib, graph_mutator, primitive_mutator)
README for additional insights
• Describes dependencies, build steps, and three stages:
a) install & load KVM-PT (kernel module)
b) build & run QEMU-PT
c) build GRUB & hypertrash_os (hypervisor_spec)
d) build & run Rust fuzzer against bhyve or QEMU targets