How thrumdev/nomt works
Main Components
• Core Database Engine – divide this into two main internal stores:
⁃ Beatree: A B-Tree–variant structure optimized for rapid random access of high-entropy key-value pairs.
⁃ Bitbox: A custom sparse binary Merkle tree stored on-disk, using a hashtable format designed for SSD optimization.
• API Layer – the key-value API that handles “Sessions” (i.e. the many-readers-one-writer transaction model).
• Supporting Libraries (from the “core” directory) – include components like hashing, page management, trie operations, proof generation, and concurrency control.
• Benchmarking and Test Harnesses:
⁃ Benchtop: A benchmarking tool housed in its own subcrate.
⁃ Fuzz: A suite for fuzz testing the API and internal methods.
⁃ Torture: An extensive testing suite for stress-testing the system.
• Experimental and Auxiliary Tools:
⁃ Trickfs/Trickmnt: A FUSE filesystem integration for deeper testing and experimental mounting of the storage layer.
• A clear separation between the external API (which provides a key-value API and “Session” based access) and the underlying database engine.
⁃ From client requests going into the API layer.
⁃ API layer interfacing with both Beatree and Bitbox in parallel.
• Feedback loops where needed—for instance, from the core library (managing pages, proofs, updates) back into the API for session coordination.
Architectural Patterns and Design Principles
• Dual-store design pattern (parallel stores: Beatree and Bitbox) that encapsulates the separation between raw key-value handling and merklized data integrity.
• “many-readers-one-writer” session management approach as part of the API’s transactional design.
• Use of modular testing and benchmarking as separate subcomponents that validate and stress the main engine.
• Mention the integration with modern hardware (SSD, NVMe, Linux as the primary target) as a performance enabler.
Relevant Technologies, Frameworks, and Languages
• Rust is the language of implementation.
• Rust components such as Cargo (as seen in the project’s Cargo.toml files) and the areas where different subcrates (nomt, core, benchtop, etc.) are used.
• Where asynchronous I/O (with Linux’s io-uring API) is being leveraged for performance.