# apache/parquet-java architecture

> Architecture diagram of the GitHub repository apache/parquet-java, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/parquet-java
- Repository: https://github.com/apache/parquet-java

## Overview

##### Project Type & Purpose
- This is a multi-module, library‐and‐toolset project for an open source, column-oriented data file format – Apache Parquet. Its primary purpose is to efficiently store, compress, encode, and process nested data structures. It supports integrations with Hadoop, Avro, Thrift, Protocol Buffers, and more.
- The repository is structured as a Maven multi-module project. Several distinct “subprojects” (modules) cover various functionalities such as core processing, encoding, integration with external systems (Hadoop, Avro, Thrift, etc.), command line interfaces, benchmarks, and plugins.

##### File Structure Analysis
- Each subproject has its own source code (usually under src/main/java), tests, and additional resources. This clearly indicates that the project is decomposed using a modular design that isolates functionalities (for example, handling column-level operations versus integration-specific code).

##### Key Insights from the README
- The README gives an overview of the purpose (managing a column-oriented file format) and lists the key features such as efficient compression, type-specific encoding, and integration with Hadoop, Avro, Thrift, and Protobuf.
- It describes the use of the “record shredding and assembly algorithm” (from the Dremel paper) to support processing of nested structures while stressing the importance of various optimizations (RLE, bit packing, etc.).

##### Main Components
- Core Modules:
– “parquet-common”: Contains common utilities, base classes, and helper functions.
– “parquet-encoding” and “parquet-column”: Implement the encoding schemes, column I/O, and data page management.
- Integration Modules:
– “parquet-hadoop”: Provides Hadoop Input/Output formats, configuration classes, and adapters for MapReduce.
– “parquet-avro”, “parquet-thrift”, “parquet-protobuf”: These modules contain support for converting between Parquet’s internal data representations and external formats such as Avro, Thrift, and Protocol Buffers.
– “parquet-cli”: Supplies command-line interfaces and tools for file inspection, conversion, and benchmarking.
– “parquet-jackson” and various “parquet-plugins”: Handle integration and optimizations such as Java Vector API support and other specialized benchmarks or vectorized encoding.
- Supporting & Utility Components:
– Build and configuration management (Maven wrapper, configuration files).
– Benchmarking, testing, and documentation modules that support quality and performance validation.

##### Relationships and Interactions
- The core modules act as the foundation—the “engine” for reading, writing, encoding, and decoding Parquet files. They expose well-defined APIs (e.g., WriteSupport, ReadSupport, RecordMaterializer) which are called by integration modules.
- Integration modules (for Hadoop, Avro, Thrift, etc.) consume the core library APIs to adapt Parquet methods to external systems. They also define plugins or extension points.
- The command-line tools and benchmarks rely on both the core modules and integration modules to provide user-facing utilities for testing and debugging.

##### Architectural Patterns and Design Principles
- Modular and layered architecture: Each module has a clear, defined responsibility. The separation between core processing and integration layers makes it easier to extend and maintain.
- Plug-in and adapter patterns: The project exposes interfaces (e.g., for record conversion) that enable pluggable implementations for different external data formats.
- Dependency inversion: Integration modules rely on abstractions provided in the core modules (often provided through well-documented APIs) rather than on concrete implementations.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Core Processing Layer"
        PC1["Parquet Common"]:::core
        PC2["Parquet Encoding"]:::core
        PC3["Parquet Column"]:::core
    end

    subgraph "Integration/Adapter Layer"
        I1["Parquet Hadoop"]:::integration
        I2["Parquet Avro"]:::integration
        I3["Parquet Thrift"]:::integration
        I4["Parquet Protobuf"]:::integration
        I5["Parquet CLI"]:::integration
        I6["Parquet Arrow"]:::integration
        I7["Parquet Jackson"]:::integration
        I8["Parquet Plugins"]:::integration
    end

    subgraph "Utility/Support Layer"
        U1["Parquet Benchmarks"]:::utility
        U2["pom.xml"]:::support
        U3[".mvn"]:::support
    end

    %% Integration Modules depend on Core Processing Layer
    I1 -->|"uses"| PC1
    I1 -->|"uses"| PC2
    I1 -->|"uses"| PC3

    I2 -->|"uses"| PC1
    I2 -->|"uses"| PC2
    I2 -->|"uses"| PC3

    I3 -->|"uses"| PC1
    I3 -->|"uses"| PC2
    I3 -->|"uses"| PC3

    I4 -->|"uses"| PC1
    I4 -->|"uses"| PC2
    I4 -->|"uses"| PC3

    I5 -->|"uses"| PC1
    I5 -->|"uses"| PC2
    I5 -->|"uses"| PC3

    I6 -->|"uses"| PC1
    I6 -->|"uses"| PC2
    I6 -->|"uses"| PC3

    I7 -->|"uses"| PC1
    I7 -->|"uses"| PC2
    I7 -->|"uses"| PC3

    I8 -->|"uses"| PC1
    I8 -->|"uses"| PC2
    I8 -->|"uses"| PC3

    %% Benchmarks rely on Core modules as well
    U1 -->|"uses"| PC1
    U1 -->|"uses"| PC2
    U1 -->|"uses"| PC3

    %% Click Events
    click PC1 "https://github.com/apache/parquet-java/tree/master/parquet-common"
    click PC2 "https://github.com/apache/parquet-java/tree/master/parquet-encoding"
    click PC3 "https://github.com/apache/parquet-java/tree/master/parquet-column"
    click I1 "https://github.com/apache/parquet-java/tree/master/parquet-hadoop"
    click I2 "https://github.com/apache/parquet-java/tree/master/parquet-avro"
    click I3 "https://github.com/apache/parquet-java/tree/master/parquet-thrift"
    click I4 "https://github.com/apache/parquet-java/tree/master/parquet-protobuf"
    click I5 "https://github.com/apache/parquet-java/tree/master/parquet-cli"
    click I6 "https://github.com/apache/parquet-java/tree/master/parquet-arrow"
    click I7 "https://github.com/apache/parquet-java/tree/master/parquet-jackson"
    click I8 "https://github.com/apache/parquet-java/tree/master/parquet-plugins"
    click U1 "https://github.com/apache/parquet-java/tree/master/parquet-benchmarks"
    click U2 "https://github.com/apache/parquet-java/blob/master/pom.xml"
    click U3 "https://github.com/apache/parquet-java/blob/master/.mvn"

    %% Styles
    classDef core fill:#FFEEAD,stroke:#333,stroke-width:2px;
    classDef integration fill:#B3E5FC,stroke:#333,stroke-width:2px;
    classDef utility fill:#C8E6C9,stroke:#333,stroke-width:2px;
    classDef support fill:#FFCDD2,stroke:#333,stroke-width:2px;
```
