# polarsignals/frostdb architecture

> Architecture diagram of the GitHub repository polarsignals/frostdb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/polarsignals/frostdb
- Repository: https://github.com/polarsignals/frostdb

## Overview

##### Main Components
- Database Core – Illustrate the central engine managing database operations.
- LSM Index and Compaction Mechanism – Show the Log-Structured Merge-tree-like index structure. Indicate the different levels (L0 in-memory parts to higher-level, compacted Parquet files) and the compaction process.
- Dynamic Schema Manager – Capture the capability for dynamic columns where new columns are created on the fly when unseen labels arrive.
- File Storage & Persistence – Model the interaction with local file system or external storage when the in-memory index rotates out.
- API/Interfaces – If applicable, include the external interfaces for client applications or the embedded usage in applications.

##### Relationships & Interactions
- Query Flow: Present how the Query Engine interacts with the LSM index and leverages Parquet’s metadata for efficient scanning, funneling results back to the client.
- Transaction Flow: Illustrate the interactions between the Transaction Manager and both the write and read paths, depicting snapshot isolation (with the watermark mechanism) to ensure consistent reads.
- Dynamic Column Updates: Indicate that when new schema elements (dynamic columns) are encountered during ingestion, the Dynamic Schema Manager updates the schema and corresponding metadata.

##### Architectural Patterns & Design Principles
- LSM Tree Pattern: Emphasize the immutable, append-only nature of data writes, organized using an LSM approach.
- Columnar Storage Architecture: Clearly mark the separation where data is stored by columns, allowing for high-performance analytic queries.
- Snapshot Isolation Mechanism: Capture the read consistency mechanism based on batching and watermarking inspired by systems like Google Spanner and Percolator.
- Modularity & Extensibility: Identify how different parts of the system (write path, query engine, schema manager) are modularized.

##### Relevant Technologies, Frameworks, and Libraries
- Go as the implementation language.
- Apache Parquet used for storage (explain its role and integration with the compaction process).
- Mention of supporting third-party tools and workflows (see .github/workflows for CI/CD pipelines).

- Parallelly, illustrate the snapshot isolation mechanism where a transaction manager sets the watermark used by the Query Engine.
- Optionally, add auxiliary modules like the Dynamic Schema Manager, external configuration & build tooling, and test suites to show overall system organization.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Nodes
    A["API/Interfaces"]:::external
    B["Database Core"]:::core
    C["Write Path - Ingestion (Arrow Records)"]:::process
    D["LSM Index & Compaction (Parquet Files)"]:::process
    E["Query Engine (Vectorized with Apache Arrow)"]:::process
    F["Dynamic Schema Manager (Dynamic Columns)"]:::process
    G["Snapshot Isolation / Transaction Manager"]:::process
    H["File Storage & Persistence (Local/External Storage)"]:::storage
    Legend["Legend:\n• Red (External): API/Interfaces\n• Blue (Core): Database Core\n• Green (Process): Write/Query/Transaction/Dynamic Schema\n• Yellow (Storage): File Storage & Persistence"]:::legend

    %% Connections
    A -->|"calls"| B
    B -->|"handles_writes"| C
    B -->|"handles_queries"| E
    B -->|"manages_txns"| G
    C -->|"ingests_data"| D
    C -->|"updates_schema"| F
    G -->|"controls_writes"| C
    G -->|"provides_snapshot"| E
    D -->|"stores_data"| H
    D -->|"scans_metadata"| E

    %% Click Events for Components
    click A "https://github.com/polarsignals/frostdb/blob/main/cmd/parquet-reencode/main.go"
    click A "https://github.com/polarsignals/frostdb/blob/main/cmd/parquet-tool/main.go"
    click B "https://github.com/polarsignals/frostdb/blob/main/db.go"
    click C "https://github.com/polarsignals/frostdb/blob/main/internal/records/record_builder.go"
    click C "https://github.com/polarsignals/frostdb/blob/main/parts/arrow.go"
    click D "https://github.com/polarsignals/frostdb/blob/main/design/lsm.md"
    click D "https://github.com/polarsignals/frostdb/blob/main/index/lsm.go"
    click E "https://github.com/polarsignals/frostdb/blob/main/query/engine.go"
    click E "https://github.com/polarsignals/frostdb/tree/main/query/physicalplan/"
    click F "https://github.com/polarsignals/frostdb/blob/main/dynparquet/dynamiccolumns.go"
    click G "https://github.com/polarsignals/frostdb/blob/main/design/snapshots.md"
    click G "https://github.com/polarsignals/frostdb/blob/main/snapshot.go"
    click H "https://github.com/polarsignals/frostdb/blob/main/storage/bucket.go"

    %% Styles
    classDef core fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef process fill:#ABEBC6,stroke:#186A3B,stroke-width:2px;
    classDef storage fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef external fill:#F5B7B1,stroke:#922B21,stroke-width:2px;
    classDef legend fill:#EAECEE,stroke:#839192,stroke-width:1px,stroke-dasharray: 5 5;
```
