How wavii/darner works
Main components
• Client Interface – Represents the external clients that send requests to the queue server using the memcache protocol (the same protocol used by Kestrel).
• Network Server – This component is implemented in the “src/net” and “include/darner/net” directories. It accepts incoming connections on port 22133 (as described in the README), manages network I/O, and processes commands coming in from clients.
• Queue Manager – Located in “src/queue” and “include/darner/queue,” this component implements the core FIFO queue functionality including the two-phase reliable fetch where messages are processed reliably.
• Logging & Utilities – Distributed across “src/util” and “include/darner/util,” this module handles logging, stats, and other auxiliary functions used throughout the server.
• Build & Deployment – Comprising files like CMakeLists.txt, the “bin/make_dist.sh” script, and related documentation in “docs”, this represents how the system is built, tested, and deployed.
• Benchmarking and Tests – The “bench” and “tests” directories include performance benchmarks and test cases that validate behavior. These aren’t part of the runtime architecture but are essential for ensuring how the components interact under load.
Relationships and interactions
• Clients communicate with the Network Server via the memcache protocol.
• The Network Server parses client requests and delegates them to the Queue Manager.
• The Queue Manager processes queue operations (push, fetch, delete, etc.) and coordinates with the underlying persistence layer (which leverages external libraries like LevelDB and Snappy for on-disk storage, as per the README).
• Utility modules such as logging and stats capture operational information, feeding back to the server for diagnostics.
• Benchmark tests and scripts are used to evaluate throughput and latency by simulating client operations.
Important architectural patterns or design principles
• Separation of Concerns – The modular structure (net, queue, util) makes clear separation between the network layer and the queue management logic.
• Out-of-Process Data Storage – Instead of holding data in RAM, Darner is designed to handle message persistence externally, integrating with disk-based storage systems (leveraging log-structured storage concepts).
• Simplicity – The “No talking! Shhh!” approach emphasizes minimal configuration and feature set for high performance and low overhead.
Relevant technologies and frameworks
• CMake – For building and managing the project.
• Boost (and potentially other C++ libraries) – Likely used to support concurrency or advanced data structures although not explicitly seen, it is required by the README.
• LevelDB and Snappy – External dependencies to handle on-disk storage with compression/fast access.
• POSIX Sockets or equivalent – Underlying network communication in the “net” modules.