# xitongsys/parquet-go architecture

> Architecture diagram of the GitHub repository xitongsys/parquet-go, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xitongsys/parquet-go
- Repository: https://github.com/xitongsys/parquet-go

## Overview

##### Main Components
- Core Libraries / Modules:
- Common: Contains shared utilities and common functions used across the project.
- Compress: Implements different compression codecs (e.g., gzip, LZ4, Snappy, ZSTD).
- Encoding: Handles binary read/write encoding for data conversion.
- Marshal/Unmarshal and Types: Responsible for converting between Go types and the Parquet binary format.
- Schema: Manages schema definition and validation with support for multiple methods (tags, JSON, etc.).
- Parquet: Contains core definitions and constants related to the Parquet file format.
- Writer & Reader: These modules provide the API interfaces for writing and reading Parquet files. They encapsulate high-level operations over the lower-level modules.
- Examples:
- A set of usage examples that demonstrate various functionalities (flat vs. nested schemas, type aliasing, JSON/CSV conversion, etc.) and illustrate how the library is used in practice.
- Tool:
- Contains command-line utilities (parquet-tools) to inspect and interact with Parquet files.
- Layout:
- Likely handles the organization of data blocks such as chunks, pages, and row groups.
- Source:
- Abstracts file operations (via the ParquetFile interface) to support various storage backends (local, HDFS, S3, etc.).

##### Relationships and Interactions
- The Encoder and Compressor modules are used by the Writer to process the data before writing.
- Conversely, during reading, Readers act upon data by decoding and decompressing it, leveraging the Encoding and Compress modules.
- The "common" module underpins utility functions that are referenced across many components, ensuring consistency (e.g., name conversions in schema handling).
- External dependencies: The ParquetFile interface (in the project’s source) indicates that the library is designed to be extended to various storage backends; this interface will be a key integration point between the core library and external file systems.
- The Example and Tool directories serve as “consumer” layers that use the APIs provided by the core modules for real-world applications and debugging.

##### Architectural Patterns and Design Principles
- Layered Architecture: There is a clear separation between low-level processing (encoding, compressing, common utilities) and high-level operations (writing, reading, and schema conversion).
- Modularity & Separation of Concerns: Each directory encapsulates a specific function (for example, “compress” for codecs and “encoding” for binary data handling).
- Extensibility: The design makes use of interfaces (e.g., ParquetFile) and well-defined abstractions so that new formats or storage solutions can be added easily.
- Concurrency: The library explicitly supports concurrent reading and writing (through parameters such as np in the initialization functions) to maximize performance.

##### Relevant Technologies and Frameworks
- Go Modules: Evidenced by go.mod and go.sum files.
- Testing: The presence of test files (ending in _test.go) across modules indicates integrated testing.
- Continuous Integration: CI configuration using Travis (.travis.yml) suggests automated testing and building.
- Command-Line Tools: The tool/parquet-tools directory illustrates command-line interface development in Go.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Utility Layer
    subgraph "Utility Layer"
        Common["Common - shared utilities"]
        Layout["Layout - data organization"]
        Types["Types - Go types for Parquet"]
    end

    %% Core Processing Layer
    subgraph "Core Processing"
        Compress["Compress - compression codecs"]
        Encoding["Encoding - binary encoding"]
        Marshal["Marshal/Unmarshal - conversion functions"]
        Schema["Schema - schema definitions"]
        Parquet["Parquet - core constants and definitions"]
        Source["Source - file abstractions"]
    end

    %% API Interface Layer
    subgraph "API Interface"
        Writer["Writer - Parquet write API"]
        Reader["Reader - Parquet read API"]
    end

    %% Consumer Layer
    subgraph "Consumer Layer"
        Examples["Examples - usage demonstrations"]
        Tool["Tool - command-line utilities"]
    end

    %% Relationships: Writer interactions
    Writer -->|"uses"| Encoding
    Writer -->|"uses"| Compress
    Writer -->|"maps_with"| Marshal
    Writer -->|"validates_with"| Schema
    Writer -->|"writes_to"| Source

    %% Relationships: Reader interactions
    Reader -->|"uses"| Encoding
    Reader -->|"uses"| Compress
    Reader -->|"maps_with"| Marshal
    Reader -->|"validates_with"| Schema
    Reader -->|"reads_from"| Source

    %% Interaction between Schema and Marshal/Unmarshal
    Schema -->|"connects"| Marshal
    Marshal -->|"supports"| Schema

    %% Utility dependencies from core modules
    Compress -->|"ref"| Common
    Encoding -->|"ref"| Common
    Marshal -->|"ref"| Common
    Schema -->|"ref"| Common
    Layout -->|"ref"| Common

    %% Click Events
    click Common "https://github.com/xitongsys/parquet-go/blob/master/common/common.go"
    click Layout "https://github.com/xitongsys/parquet-go/tree/master/layout/"
    click Types "https://github.com/xitongsys/parquet-go/blob/master/types/types.go"
    click Compress "https://github.com/xitongsys/parquet-go/tree/master/compress/"
    click Encoding "https://github.com/xitongsys/parquet-go/tree/master/encoding/"
    click Marshal "https://github.com/xitongsys/parquet-go/tree/master/marshal/"
    click Schema "https://github.com/xitongsys/parquet-go/blob/master/schema/schemahandler.go"
    click Parquet "https://github.com/xitongsys/parquet-go/tree/master/parquet/"
    click Source "https://github.com/xitongsys/parquet-go/blob/master/source/source.go"
    click Writer "https://github.com/xitongsys/parquet-go/blob/master/writer/writer.go"
    click Reader "https://github.com/xitongsys/parquet-go/blob/master/reader/reader.go"
    click Examples "https://github.com/xitongsys/parquet-go/tree/master/example/"
    click Tool "https://github.com/xitongsys/parquet-go/tree/master/tool/parquet-tools/"

    %% Styles
    classDef utility fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef core fill:#FCF3CF,stroke:#7D6608,stroke-width:2px;
    classDef api fill:#D5F5E3,stroke:#196F3D,stroke-width:2px;
    classDef consumer fill:#F5B7B1,stroke:#943126,stroke-width:2px;

    class Common,Layout,Types utility;
    class Compress,Encoding,Marshal,Schema,Parquet,Source core;
    class Writer,Reader api;
    class Examples,Tool consumer;
```
