# hoglandets-it/beetl architecture

> Architecture diagram of the GitHub repository hoglandets-it/beetl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hoglandets-it/beetl
- Repository: https://github.com/hoglandets-it/beetl

## Overview

##### Main Components

a. Core Components:
- ETL Engine (Beetl Core)
- Configuration Handler
- Source Connectors
- Transformers
- Result Handler

b. Data Sources/Destinations (based on the /src/beetl/sources directory):
- SQL Databases (MySQL, PostgreSQL, SQL Server)
- MongoDB
- REST API
- XML
- CSV
- Static Data
- Faker (for testing)
- iTop

**┌─ Configuration Layer ─────────────────────┐**
- YAML/JSON/Python Dict Configuration     │
- Column Definitions                      │
- Sync Rules                             │
- Core Processing Layer ──────────────────┤
- ┌─ Source Layer ┐    ┌─ Transform Layer ┐│
- Connectors  │ -> │ - Data Frames    ││
- Extractors  │    │ - Type Convert   ││
- ┘    └────────────────┬─┘│

- ┌─ Destination Layer ───────────────┐   │
- Target Systems                  │<──┘
- Data Loading                    │    │
- ┘    │
- ┘

##### Key Elements to Highlight

a. Source Connectors:
- Them by type (SQL, NoSQL, API, File-based)
- Interface inheritance from base connector

b. Transformation Pipeline:
- Data flow through transformers
- Supported transformation types:
- Frames
- Integer operations
- String operations
- Regular expressions
- Structural transformations
- iTop-specific transformations

c. Configuration Flow:
- How configuration is parsed
- Validation process
- Demonstrate how it connects to source/destination selection

##### Specific Implementation Details

Technical aspects:
- Polars integration for data processing
- Python class hierarchy
- Async/sync operation support
- Error handling flow
- Validation checkpoints

##### External Dependencies

Connections to:
- Polars for data processing
- Database drivers
- API clients
- XML processors

##### Testing Infrastructure

A separate section showing:
- Integration testing framework
- Benchmark testing setup
- Unit testing components
- Test containers

##### Deployment Options

Deployment possibilities:
- Docker container
- Kubernetes deployment
- Direct Python package installation

For this specific ETL tool, focus on showing the clear separation between configuration, extraction, transformation, and loading phases, while highlighting the extensibility points that allow for adding new source types and transformers.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Configuration["Configuration Layer"]
        Config[Configuration Handler]:::config
        YAML[YAML/JSON/Python Dict]:::config
        ColDef[Column Definitions]:::config
        SyncRules[Sync Rules]:::config
        
        YAML --> Config
        ColDef --> Config
        SyncRules --> Config
    end

    subgraph Core["Core Processing Layer"]
        Engine[ETL Engine Core]:::core
        
        subgraph Sources["Source Layer"]
            direction TB
            SourceInterface[Source Interface]:::interface
            
            subgraph SQL["SQL Sources"]
                MySQL[MySQL]:::source
                PostgreSQL[PostgreSQL]:::source
                SQLServer[SQL Server]:::source
            end
            
            subgraph NoSQL["NoSQL Sources"]
                MongoDB[MongoDB]:::source
            end
            
            subgraph API["API Sources"]
                REST[REST API]:::source
                iTopSrc[iTop]:::source
            end
            
            subgraph Files["File Sources"]
                XML[XML]:::source
                CSV[CSV]:::source
                Static[Static Data]:::source
                Faker[Faker]:::source
            end
        end

        subgraph Transform["Transform Layer"]
            TransInterface[Transformer Interface]:::interface
            
            Frames[Frame Operations]:::transform
            Integer[Integer Operations]:::transform
            Strings[String Operations]:::transform
            Regex[Regular Expressions]:::transform
            Structs[Structural Operations]:::transform
            iTopTrans[iTop Transformations]:::transform
            Misc[Miscellaneous]:::transform
        end

        subgraph Destination["Destination Layer"]
            ResultHandler[Result Handler]:::destination
        end
    end

    subgraph Testing["Testing Infrastructure"]
        IntegrationTests[Integration Tests]:::test
        UnitTests[Unit Tests]:::test
        BenchmarkTests[Benchmark Tests]:::test
        TestContainers[Test Containers]:::test
    end

    subgraph Deployment["Deployment Options"]
        Docker[Docker]:::deploy
        Kubernetes[Kubernetes]:::deploy
    end

    %% Relationships
    Config --> Engine
    Engine --> SourceInterface
    SourceInterface --> Transform
    Transform --> ResultHandler

    %% Click events
    click Config "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/config.py"
    click Engine "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/beetl.py"
    click ResultHandler "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/result.py"
    click SourceInterface "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/sources/interface.py"
    click MySQL "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/sources/mysql.py"
    click PostgreSQL "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/sources/postgresql.py"
    click SQLServer "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/sources/sqlserver.py"
    click MongoDB "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/sources/mongodb.py"
    click REST "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/sources/rest.py"
    click XML "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/sources/xml.py"
    click CSV "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/sources/csv.py"
    click Static "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/sources/static.py"
    click Faker "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/sources/faker.py"
    click iTopSrc "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/sources/itop.py"
    click TransInterface "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/transformers/interface.py"
    click Frames "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/transformers/frames.py"
    click Integer "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/transformers/integer.py"
    click Strings "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/transformers/strings.py"
    click Regex "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/transformers/regex.py"
    click Structs "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/transformers/structs.py"
    click iTopTrans "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/transformers/itop.py"
    click Misc "https://github.com/hoglandets-it/beetl/blob/develop/src/beetl/transformers/misc.py"
    click Docker "https://github.com/hoglandets-it/beetl/tree/develop/docker/Dockerfile"
    click Kubernetes "https://github.com/hoglandets-it/beetl/tree/develop/kubernetes"
    click IntegrationTests "https://github.com/hoglandets-it/beetl/tree/develop/tests/integration_testing"
    click UnitTests "https://github.com/hoglandets-it/beetl/tree/develop/tests/unit_testing"
    click BenchmarkTests "https://github.com/hoglandets-it/beetl/tree/develop/tests/benchmark_testing"
    click TestContainers "https://github.com/hoglandets-it/beetl/tree/develop/tests/project_testcontainers"

    %% Styling
    classDef core fill:#2196F3,stroke:#1565C0,color:white
    classDef config fill:#9E9E9E,stroke:#616161,color:white
    classDef source fill:#4CAF50,stroke:#2E7D32,color:white
    classDef transform fill:#FFC107,stroke:#FFA000,color:black
    classDef destination fill:#FF9800,stroke:#F57C00,color:white
    classDef interface fill:#90CAF9,stroke:#64B5F6,color:black
    classDef test fill:#E91E63,stroke:#C2185B,color:white
    classDef deploy fill:#673AB7,stroke:#512DA8,color:white
```
