# apache/iceberg-python architecture

> Architecture diagram of the GitHub repository apache/iceberg-python, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apache/iceberg-python
- Repository: https://github.com/apache/iceberg-python

## Overview

##### Main Components

a. Client Interface Layer
- CLI Component (pyiceberg/cli/)
- Public API Interface (top-level modules like schema.py, types.py)

b. Core Processing Layer
- Table Operations (pyiceberg/table/)
- Expression Processing (pyiceberg/expressions/)
- Schema Management (pyiceberg/schema.py)
- Type System (pyiceberg/types.py)

c. Storage & Catalog Layer
- Catalog Implementations (pyiceberg/catalog/)
- DynamoDB
- Glue
- Hive
- REST
- SQL
- I/O Operations (pyiceberg/io/)
- FSSpec
- PyArrow

d. Data Format Layer
- Avro Implementation (pyiceberg/avro/)
- Codecs
- Encoding/Decoding
- File Operations

##### Component Relationships

Key interactions:
**Client Layer**
- ↓
**Core Processing Layer**
- ↓
**Storage & Catalog Layer ←→ External Storage Systems**
- ↓
**Data Format Layer**

##### Important Details

a. Integration Points:
- External catalog systems (Hive, DynamoDB, etc.)
- Storage backends (S3, local filesystem)
- Data format handling (Avro)

b. Key Features:
- Table metadata management
- Schema evolution
- Partition handling
- Expression processing
- Statistics tracking

##### Implementation Notes

- Related components together
- Notes for key configuration points
- Version compatibility information

##### Technical Details

- Support for different file formats
- Catalog implementation options
- I/O abstractions
- Expression evaluation flow
- Schema management capabilities

Remember to include:
- Version information
- Required dependencies
- Configuration options
- Integration points
- Error handling flows
- Data flow directions

This will provide a comprehensive view of PyIceberg's architecture while maintaining clarity and usefulness for technical discussions.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Style definitions
    classDef client fill:#3498db,stroke:#2980b9,color:white
    classDef core fill:#2ecc71,stroke:#27ae60,color:white
    classDef storage fill:#e67e22,stroke:#d35400,color:white
    classDef format fill:#9b59b6,stroke:#8e44ad,color:white
    classDef external fill:#95a5a6,stroke:#7f8c8d,color:white

    %% Client Interface Layer
    subgraph ClientLayer
        CLI["CLI Interface"]:::client
        API["Public API"]:::client
    end
    click CLI "https://github.com/apache/iceberg-python/tree/main/pyiceberg/cli/"
    click API "https://github.com/apache/iceberg-python/blob/main/pyiceberg/schema.py"

    %% Core Processing Layer
    subgraph CoreLayer
        TableOps["Table Operations"]:::core
        ExprProc["Expression Processing"]:::core
        SchemaMan["Schema Management"]:::core
        TypeSys["Type System"]:::core
    end
    click TableOps "https://github.com/apache/iceberg-python/tree/main/pyiceberg/table/"
    click ExprProc "https://github.com/apache/iceberg-python/tree/main/pyiceberg/expressions/"
    click SchemaMan "https://github.com/apache/iceberg-python/blob/main/pyiceberg/schema.py"
    click TypeSys "https://github.com/apache/iceberg-python/blob/main/pyiceberg/types.py"

    %% Storage & Catalog Layer
    subgraph StorageLayer
        subgraph Catalogs
            DynamoDB["DynamoDB Catalog"]:::storage
            Glue["Glue Catalog"]:::storage
            Hive["Hive Catalog"]:::storage
            REST["REST Catalog"]:::storage
            SQL["SQL Catalog"]:::storage
        end
        subgraph IO
            FSSpec["FSSpec"]:::storage
            PyArrow["PyArrow"]:::storage
        end
    end
    click DynamoDB "https://github.com/apache/iceberg-python/blob/main/pyiceberg/catalog/dynamodb.py"
    click Glue "https://github.com/apache/iceberg-python/blob/main/pyiceberg/catalog/glue.py"
    click Hive "https://github.com/apache/iceberg-python/blob/main/pyiceberg/catalog/hive.py"
    click REST "https://github.com/apache/iceberg-python/blob/main/pyiceberg/catalog/rest.py"
    click SQL "https://github.com/apache/iceberg-python/blob/main/pyiceberg/catalog/sql.py"
    click FSSpec "https://github.com/apache/iceberg-python/blob/main/pyiceberg/io/fsspec.py"
    click PyArrow "https://github.com/apache/iceberg-python/blob/main/pyiceberg/io/pyarrow.py"

    %% Data Format Layer
    subgraph FormatLayer
        subgraph AvroImpl["Avro Implementation"]
            Codecs["Codecs"]:::format
            FileOps["File Operations"]:::format
            Reader["Reader"]:::format
            Writer["Writer"]:::format
            Decoder["Decoder"]:::format
            Encoder["Encoder"]:::format
        end
    end
    click Codecs "https://github.com/apache/iceberg-python/tree/main/pyiceberg/avro/codecs/"
    click FileOps "https://github.com/apache/iceberg-python/blob/main/pyiceberg/avro/file.py"
    click Reader "https://github.com/apache/iceberg-python/blob/main/pyiceberg/avro/reader.py"
    click Writer "https://github.com/apache/iceberg-python/blob/main/pyiceberg/avro/writer.py"
    click Decoder "https://github.com/apache/iceberg-python/blob/main/pyiceberg/avro/decoder.py"
    click Encoder "https://github.com/apache/iceberg-python/blob/main/pyiceberg/avro/encoder.py"

    %% Table Management Components
    subgraph TableManagement
        MetaMan["Metadata Management"]:::core
        Stats["Statistics"]:::core
        Snaps["Snapshots"]:::core
        SchemaUpd["Schema Updates"]:::core
        Sort["Sorting"]:::core
    end
    click MetaMan "https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/metadata.py"
    click Stats "https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/statistics.py"
    click Snaps "https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/snapshots.py"
    click SchemaUpd "https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/update/schema.py"
    click Sort "https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/sorting.p"

    %% Relationships
    CLI --> API
    API --> CoreLayer
    CoreLayer --> StorageLayer
    StorageLayer --> FormatLayer
    TableManagement --> CoreLayer
    IO --> ExternalStorage[("External Storage Systems")]:::external

    %% Legend
    subgraph Legend
        Client["Client Interface"]:::client
        Core["Core Processing"]:::core
        Storage["Storage & Catalog"]:::storage
        Format["Data Format"]:::format
        External["External Systems"]:::external
    end
```
