How wiredtiger/wiredtiger works
• API/Connection Interface: At the top layer, the storage engine exposes its API (see “src/include/api.h”, “src/conn/conn_api.c”, etc.). This is where the application or upper layers interface with WiredTiger.
• Session and Transaction Management: Modules in “src/session”, “src/txn”, and “src/conf” constitute the management of sessions, transactions, and configurations.
• Data Storage & B-tree Structures: The “src/btree” and “src/schema” directories implement the core data structures. They handle B-tree page handling, row and column key/value storage, and schema management.
• Block and Cache Management: The “src/block” and “src/block_cache” directories show how data is abstracted as blocks and how caching, chunk-caches, and buffer management are implemented.
• Logging and Durability: Modules under “src/log” (with related Doxygen files in “src/docs”) indicate the logging subsystem that maintains durability, supports checkpoints, and handles recovery.
• Checkpoint, Recovery, and Rollback: The “src/checkpoint”, “src/rollback_to_stable”, and parts of “src/reconcile” manage periodic persistence of data and recovery consistency.
• Compression and Encryption Plugins: The “ext/compressors” and “ext/encryptors” directories indicate a plugin-style architecture where multiple algorithms can be selected at build time or dynamically configured.
• Storage Extensions and Tiered Storage: The “ext/storage_sources” (with subdirectories like “azure_store”, “gcp_store”, “s3_store”) and the “src/tiered” module suggest external storage integrations and tiered storage support.
• Utility and Support Functions: Finally, the “src/support” and “src/utilities” directories include miscellaneous functions for error handling, configuration parsing, and internal diagnostics.
Component Relationships and Data Flow
• Session/Connection Layer: Show a “Session Manager” component that manages connections and converts API calls into lower-level operations.
– B-tree / Schema for data manipulation.
– Block I/O and Cache Manager that manage on-disk representation.
– Logging subsystem that records changes for durability.
• Checkpoint/Recovery and Rollback: Illustrate how the logging component feeds into the checkpoint module and how rollback/recovery components interact with B-tree structures to ensure consistency.
• Utilities: Optionally, add a lower support layer that provides auxiliary functions (error processing, configuration, multi-threading support) that are used by multiple modules.
Architectural Patterns and Design Decisions
• Layered/Modular Architecture: Indicate the layering from the external API down to the low-level disk access and plugin integrations.
• Plug-in Mechanism: Emphasize how compression, encryption, and tiered storage components are separate from core logic, making WiredTiger highly modular.
• Concurrent and Fault-tolerant Design: Note how session management, logging, and checkpoint modules collaborate to ensure transactional integrity and data durability.