How polarsignals/frostdb works
Main Components
• Database Core – Illustrate the central engine managing database operations.
• LSM Index and Compaction Mechanism – Show the Log-Structured Merge-tree-like index structure. Indicate the different levels (L0 in-memory parts to higher-level, compacted Parquet files) and the compaction process.
• Dynamic Schema Manager – Capture the capability for dynamic columns where new columns are created on the fly when unseen labels arrive.
• File Storage & Persistence – Model the interaction with local file system or external storage when the in-memory index rotates out.
• API/Interfaces – If applicable, include the external interfaces for client applications or the embedded usage in applications.
Relationships & Interactions
• Query Flow: Present how the Query Engine interacts with the LSM index and leverages Parquet’s metadata for efficient scanning, funneling results back to the client.
• Transaction Flow: Illustrate the interactions between the Transaction Manager and both the write and read paths, depicting snapshot isolation (with the watermark mechanism) to ensure consistent reads.
• Dynamic Column Updates: Indicate that when new schema elements (dynamic columns) are encountered during ingestion, the Dynamic Schema Manager updates the schema and corresponding metadata.
Architectural Patterns & Design Principles
• LSM Tree Pattern: Emphasize the immutable, append-only nature of data writes, organized using an LSM approach.
• Columnar Storage Architecture: Clearly mark the separation where data is stored by columns, allowing for high-performance analytic queries.
• Snapshot Isolation Mechanism: Capture the read consistency mechanism based on batching and watermarking inspired by systems like Google Spanner and Percolator.
• Modularity & Extensibility: Identify how different parts of the system (write path, query engine, schema manager) are modularized.
Relevant Technologies, Frameworks, and Libraries
• Go as the implementation language.
• Apache Parquet used for storage (explain its role and integration with the compaction process).
• Mention of supporting third-party tools and workflows (see .github/workflows for CI/CD pipelines).
• Parallelly, illustrate the snapshot isolation mechanism where a transaction manager sets the watermark used by the Query Engine.
• Optionally, add auxiliary modules like the Dynamic Schema Manager, external configuration & build tooling, and test suites to show overall system organization.