How jimyag/parquet-tools works
Main Components
• Main Application Entry: Represented by main.go, which serves as the application’s starting point.
• Command Dispatcher & Subcommands: The “cmd” directory is composed of several files (cat.go, diff.go, footer.go, meta.go, schema.go, struct.go, and root.go) that implement the different CLI commands. Each file can be depicted as a separate component that handles the corresponding functionality.
• Internal Modules:
– Reader: Located in internal/reader, with separate implementations for HTTP (reader_http.go) and S3 (reader_s3.go). This module abstracts file retrieval from various sources.
– Dumper: Located in internal/dumper, responsible for formatting and printing the inspection output.
Component Interactions
• The Application Flow begins when the user calls the “parquet-tools” binary through the command line.
• The main application (main.go) delegates to a command dispatcher (cmd/root.go) which then routes the control flow to the specific subcommand file (e.g., cat.go for printing records, meta.go for metadata inspection).
• The selected subcommand then calls the appropriate internal reader module (either the HTTP reader or S3 reader, or a direct local file reader) to read the Parquet file content.
• Once the file is read, the data is passed to the dumper module (internal/dumper) to format and output the result accordingly (pretty printing JSON, Avro schema, or generating a Go struct).
• Optionally, the configuration data (like the s3 TOML configuration) is used when accessing S3 resources.
• Additionally, for the “diff” command, the tool likely uses two separate reader calls to load both schemas and then a comparison logic that would be part of the command’s logic.
Architectural Patterns and Design Principles
• Separation of Concerns: CLI command logic is separated from the actual business logic (reading and dump/format operations).
• Modular Design: The internal packages (reader and dumper) encapsulate specific responsibilities, making the system easier to extend if new protocols or output formats are added.
• Interface Abstraction (implied): Although not explicit in the file tree, the design suggests that there could be an interface for file readers that abstracts over different protocols (HTTP, S3, local).