# jimyag/parquet-tools architecture

> Architecture diagram of the GitHub repository jimyag/parquet-tools, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jimyag/parquet-tools
- Repository: https://github.com/jimyag/parquet-tools

## Overview

##### Main Components
- Main Application Entry: Represented by main.go, which serves as the application’s starting point.
- Command Dispatcher & Subcommands: The “cmd” directory is composed of several files (cat.go, diff.go, footer.go, meta.go, schema.go, struct.go, and root.go) that implement the different CLI commands. Each file can be depicted as a separate component that handles the corresponding functionality.
- Internal Modules:
– Reader: Located in internal/reader, with separate implementations for HTTP (reader_http.go) and S3 (reader_s3.go). This module abstracts file retrieval from various sources.
– Dumper: Located in internal/dumper, responsible for formatting and printing the inspection output.

##### Component Interactions
- The Application Flow begins when the user calls the “parquet-tools” binary through the command line.
- The main application (main.go) delegates to a command dispatcher (cmd/root.go) which then routes the control flow to the specific subcommand file (e.g., cat.go for printing records, meta.go for metadata inspection).
- The selected subcommand then calls the appropriate internal reader module (either the HTTP reader or S3 reader, or a direct local file reader) to read the Parquet file content.
- Once the file is read, the data is passed to the dumper module (internal/dumper) to format and output the result accordingly (pretty printing JSON, Avro schema, or generating a Go struct).
- Optionally, the configuration data (like the s3 TOML configuration) is used when accessing S3 resources.
- Additionally, for the “diff” command, the tool likely uses two separate reader calls to load both schemas and then a comparison logic that would be part of the command’s logic.

##### Architectural Patterns and Design Principles
- Separation of Concerns: CLI command logic is separated from the actual business logic (reading and dump/format operations).
- Modular Design: The internal packages (reader and dumper) encapsulate specific responsibilities, making the system easier to extend if new protocols or output formats are added.
- Interface Abstraction (implied): Although not explicit in the file tree, the design suggests that there could be an interface for file readers that abstracts over different protocols (HTTP, S3, local).

## Diagram (Mermaid)

```mermaid
flowchart TD
    UserInput("User CLI Input"):::external
    UserInput -->|"initiates"| MainApp("Main Application (main.go)"):::cli
    MainApp -->|"delegates-to"| CmdDispatcher("Command Dispatcher (cmd/root.go)"):::cli

    subgraph "CLI Commands"
        Cat("Cat Command (cmd/cat.go)"):::cli
        Diff("Diff Command (cmd/diff.go)"):::cli
        Footer("Footer Command (cmd/footer.go)"):::cli
        Meta("Meta Command (cmd/meta.go)"):::cli
        Schema("Schema Command (cmd/schema.go)"):::cli
        Struct("Struct Command (cmd/struct.go)"):::cli
    end

    CmdDispatcher -->|"routes-to"| Cat
    CmdDispatcher -->|"routes-to"| Diff
    CmdDispatcher -->|"routes-to"| Footer
    CmdDispatcher -->|"routes-to"| Meta
    CmdDispatcher -->|"routes-to"| Schema
    CmdDispatcher -->|"routes-to"| Struct

    subgraph "Internal Modules"
        subgraph "Reader Module"
            HTTPReader("HTTP Reader (internal/reader/reader_http.go)"):::internal
            S3Reader("S3 Reader (internal/reader/reader_s3.go)"):::internal
            LocalReader("Local File Reader"):::internal
        end
        Dumper("Dumper (internal/dumper/dumper.go)"):::internal
    end

    ParquetSource("Parquet File Source"):::external
    Config("Configuration (s3.toml)"):::external
    Output("Output (Console)"):::external

    %% Commands invoking a reader (local by default)
    Cat -->|"invokes"| LocalReader
    Meta -->|"invokes"| LocalReader
    Footer -->|"invokes"| LocalReader
    Schema -->|"invokes"| LocalReader
    Struct -->|"invokes"| LocalReader
    Diff -->|"invokes"| LocalReader

    %% Additional protocol choices for Diff command
    Diff -->|"may-invoke"| HTTPReader
    Diff -->|"may-invoke"| S3Reader

    %% Reader module connections
    LocalReader -->|"reads-from"| ParquetSource
    HTTPReader -->|"reads-from"| ParquetSource
    S3Reader -->|"reads-from"| ParquetSource

    %% Configuration feeds S3 Reader
    Config -->|"supplies"| S3Reader

    %% Reader passes data to Dumper
    LocalReader -->|"sends-data"| Dumper
    HTTPReader -->|"sends-data"| Dumper
    S3Reader -->|"sends-data"| Dumper

    Dumper -->|"outputs-to"| Output

    %% Click Events
    click MainApp "https://github.com/jimyag/parquet-tools/blob/main/main.go"
    click CmdDispatcher "https://github.com/jimyag/parquet-tools/blob/main/cmd/root.go"
    click Cat "https://github.com/jimyag/parquet-tools/blob/main/cmd/cat.go"
    click Diff "https://github.com/jimyag/parquet-tools/blob/main/cmd/diff.go"
    click Footer "https://github.com/jimyag/parquet-tools/blob/main/cmd/footer.go"
    click Meta "https://github.com/jimyag/parquet-tools/blob/main/cmd/meta.go"
    click Schema "https://github.com/jimyag/parquet-tools/blob/main/cmd/schema.go"
    click Struct "https://github.com/jimyag/parquet-tools/blob/main/cmd/struct.go"
    click HTTPReader "https://github.com/jimyag/parquet-tools/blob/main/internal/reader/reader_http.go"
    click S3Reader "https://github.com/jimyag/parquet-tools/blob/main/internal/reader/reader_s3.go"
    click Dumper "https://github.com/jimyag/parquet-tools/blob/main/internal/dumper/dumper.go"

    %% Styling
    classDef cli fill:#cce5ff,stroke:#333,stroke-width:2px;
    classDef internal fill:#d4edda,stroke:#28a745,stroke-width:2px;
    classDef external fill:#f8d7da,stroke:#dc3545,stroke-width:2px;
```
