# prometheus-community/parquet-common architecture

> Architecture diagram of the GitHub repository prometheus-community/parquet-common, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/prometheus-community/parquet-common
- Repository: https://github.com/prometheus-community/parquet-common

## Overview

##### Project Type & Purpose
- This is a shared Go library (not a standalone service) aimed at providing
– a common Parquet schema for time-series data
– high-performance encoding/decoding
– utilities for querying and block indexing
- Intended consumers: Cortex, Thanos, other Prometheus-compatible systems.

##### File Structure
Top-level packages:
- schema/      → defines Parquet schema, builders, encoder logic
- convert/     → transforms Go types into Parquet row groups, merges blocks, writer abstraction
- storage/     → Parquet shard management, random-access reads
- search/      → constraint definitions, materialization of row filters, row-range logic
- queryable/   → high-level API to run queries against Parquet files
- util/        → shared helpers (fixtures, string utilities)
- CI workflows, Makefile, go.mod → build/test/benchmark orchestration

##### Extract Architectural Insights from README
- Core components: schema definitions + encoder/decoder + query utilities
- Integration with object stores (S3/GCS/Azure) for long-term storage
- Reusable Go types, logical-type hints, block indexing

##### Main Components & Their Relationships
A. schema package
– Defines the Parquet data model & builder
– Exposes Go structs that map 1:1 to Parquet logical types
B. convert package
– Uses schema to encode time-series points into Parquet row-groups
– Writer interface abstracts file vs. in-memory writes
– merge logic compacts multiple row-groups
C. storage package
– Manages Parquet “shards” (files) on object stores
– read_at for random reads needed by query layer
D. search package
– Applies query constraints against block indexes
– Determines row ranges to materialize
E. queryable package
– High-level entry point: given a Parquet file, returns an iterator/queryable
– Leverages storage.read_at and search filters
F. util package
– Common helper functions used across all other modules
G. External Systems
– Object Store (S3, GCS, Azure Blob)
– Downstream consumers (Cortex/Thanos) call into queryable

##### Architectural Patterns & Design Principles
- Modular layering: schema → convert → storage → search → queryable
- Single Responsibility: each package focuses on one concern
- Builder pattern in schema/schema_builder.go
- Abstraction of I/O via writer interfaces & read_at
- Test-driven foundation: one test file per core component

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    ObjectStore[(Object Store)]:::external
    Prometheus[/"Prometheus Consumer"/]:::external

    %% Core Modules
    subgraph "Core Library Modules"
        Schema["schema\n(Defines Parquet schema, Builder)"]:::core
        Convert["convert\n(Encoder, Writer abstraction, Merge)"]:::core
        Storage["storage\n(Shard management, read_at)"]:::core
        Search["search\n(Constraint engine, row-range)"]:::core
        Queryable["queryable\n(Iterator API)"]:::core
    end

    %% Util Layer
    Util["util\n(Helpers)"]:::util

    %% Data Flow
    Schema -->|Encode schema| Convert
    Convert -->|Write| Storage
    Storage -->|Read_at| Search
    Search -->|Filter rows| Queryable

    %% External interactions
    Convert -->|Write Parquet| ObjectStore
    Storage -->|Shard write| ObjectStore
    Queryable -->|Read Parquet| ObjectStore
    Prometheus -->|API calls| Queryable

    %% Click Events
    click Schema "https://github.com/prometheus-community/parquet-common/tree/main/schema/"
    click Convert "https://github.com/prometheus-community/parquet-common/tree/main/convert/"
    click Storage "https://github.com/prometheus-community/parquet-common/tree/main/storage/"
    click Search "https://github.com/prometheus-community/parquet-common/tree/main/search/"
    click Queryable "https://github.com/prometheus-community/parquet-common/tree/main/queryable/"
    click Util "https://github.com/prometheus-community/parquet-common/tree/main/util/"

    %% Legend Styles
    classDef core fill:#AED6F1,stroke:#21618C,stroke-width:2px;
    classDef util fill:#FDEDEC,stroke-dasharray: 5 5,stroke:#C0392B,stroke-width:2px;
    classDef external fill:#D5F5E3,stroke:#229954,stroke-width:2px;
    classDef default stroke:#000,stroke-width:1px;
```
