How prometheus-community/parquet-common works
Project Type & Purpose
• This is a shared Go library (not a standalone service) aimed at providing
– a common Parquet schema for time-series data
– high-performance encoding/decoding
– utilities for querying and block indexing
• Intended consumers: Cortex, Thanos, other Prometheus-compatible systems.
File Structure
Top-level packages:
• schema/ → defines Parquet schema, builders, encoder logic
• convert/ → transforms Go types into Parquet row groups, merges blocks, writer abstraction
• storage/ → Parquet shard management, random-access reads
• search/ → constraint definitions, materialization of row filters, row-range logic
• queryable/ → high-level API to run queries against Parquet files
• util/ → shared helpers (fixtures, string utilities)
• CI workflows, Makefile, go.mod → build/test/benchmark orchestration
Extract Architectural Insights from README
• Core components: schema definitions + encoder/decoder + query utilities
• Integration with object stores (S3/GCS/Azure) for long-term storage
• Reusable Go types, logical-type hints, block indexing
Main Components & Their Relationships
A. schema package
– Defines the Parquet data model & builder
– Exposes Go structs that map 1:1 to Parquet logical types
B. convert package
– Uses schema to encode time-series points into Parquet row-groups
– Writer interface abstracts file vs. in-memory writes
– merge logic compacts multiple row-groups
C. storage package
– Manages Parquet “shards” (files) on object stores
– read_at for random reads needed by query layer
D. search package
– Applies query constraints against block indexes
– Determines row ranges to materialize
E. queryable package
– High-level entry point: given a Parquet file, returns an iterator/queryable
– Leverages storage.read_at and search filters
F. util package
– Common helper functions used across all other modules
G. External Systems
– Object Store (S3, GCS, Azure Blob)
– Downstream consumers (Cortex/Thanos) call into queryable
Architectural Patterns & Design Principles
• Modular layering: schema → convert → storage → search → queryable
• Single Responsibility: each package focuses on one concern
• Builder pattern in schema/schema_builder.go
• Abstraction of I/O via writer interfaces & read_at
• Test-driven foundation: one test file per core component