# dlt-hub/dlt architecture

> Architecture diagram of the GitHub repository dlt-hub/dlt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dlt-hub/dlt
- Repository: https://github.com/dlt-hub/dlt

## Overview

##### Main Components

a) Core Components:
- Extract Layer (/dlt/extract)
- Load Layer (/dlt/load)
- Normalize Layer (/dlt/normalize)
- Pipeline Engine (/dlt/pipeline)
- Configuration System (/dlt/common/configuration)
- Schema Management (/dlt/common/schema)

b) Integration Points:
- Sources System (/dlt/sources)
- Destinations System (/dlt/destinations)
- Helpers System (/dlt/helpers)

**┌─ Client Layer ─────────────────────────────────────┐**
- Python Scripts                                   │
- Notebooks                                        │
- Airflow DAGs                                    │
- CLI Interface                                    │
- ┘
- ↓
**┌─ Pipeline Layer ──────────────────────────────────┐**
- ┌─ Pipeline Engine ───────────────────────────┐   │
- Pipeline Configuration                    │   │
- State Management                          │   │
- Execution Control                         │   │
- ┘ │
- ┘
- ↓
**┌─ Core Processing Layer ───────────────────────────┐**
- ┌─────────┐      ┌──────────┐      ┌──────────┐  │
- Extract │ →    │Normalize │  →   │  Load    │  │
- ┘      └──────────┘      └──────────┘  │
- ┘
- ↕
**┌─ Common Services Layer ──────────────────────────┐**
- Configuration Management                       │
- Schema Management                             │
- Storage Management                            │
- Runtime Services                              │
- ┘
- ↕
**┌─ Integration Layer ────────────────────────────────┐**
- ┌─ Sources ─┐   ┌─ Destinations ┐   ┌─ Helpers ┐  │
- REST    │   │ - DuckDB      │   │ - DBT    │  │
- SQL     │   │ - BigQuery    │   │ - Air-   │  │
- Files   │   │ - Snowflake   │   │   flow   │  │
- ┘   └──────────────┘   └──────────┘  │
- ┘

##### Important Details

a) Data Flow Indicators:
- Primary data flow through Extract → Normalize → Load
- Configuration and schema management flows
- Interaction points with external systems

b) Component Details:
- Supported source types
- Supported destination types
- Helper integrations
- Configuration points

##### Additional Elements

- Component type indicators
- Data flow line types
- Integration point symbols

b) Notes:
- Version compatibility
- Extension points
- Configuration requirements

##### Implementation Details

- Maintain consistent spacing and alignment
- Use clear, readable fonts
- A title and version information
- Brief component descriptions

##### Key Relationships to Highlight

- Pipeline orchestration flow
- Data transformation stages
- Configuration management
- Schema evolution
- State management
- Error handling paths

This system design should effectively communicate the architecture while maintaining enough detail to be useful for development and maintenance purposes.

Remember to include the project's key features:
- Automatic Schema Management
- Data Normalization
- Incremental Loading
- Multiple Destination Support
- Source Integration Framework
- Configuration Management
- CLI Integration

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph DLT["DLT System"]
        subgraph Core["Core Processing"]
            Extract["Extract Layer"]
            Normalize["Normalize Layer"]
            Load["Load Layer"]
            Pipeline["Pipeline Engine"]
            
            Extract --> Normalize
            Normalize --> Load
            Pipeline --> Extract
            Pipeline --> Normalize
            Pipeline --> Load
        end

        subgraph Common["Common Infrastructure"]
            Config["Configuration System"]
            Schema["Schema Management"]
            Storage["Storage Systems"]
            Types["Type Management"]
            Writers["Data Writers"]
            SchemaEvo["Schema Evolution"]
            State["Pipeline State"]
            Monitor["Monitoring & Telemetry"]

            Config --> Schema
            Schema --> SchemaEvo
            Storage --> Writers
            Types --> Schema
            State --> Pipeline
        end

        subgraph Integration["Integration Layer"]
            CLI["CLI Interface"]
            Helpers["Helper Systems"]
            Sources["Source Connectors"]
            Destinations["Destination Adapters"]
            Incremental["Incremental Loading"]

            CLI --> Pipeline
            Helpers --> Pipeline
            Sources --> Extract
            Load --> Destinations
            Incremental --> Extract
        end
    end

    %% Component Mappings
    click Extract "https://github.com/dlt-hub/dlt/blob/devel/dlt/extract/extract.py"
    click Load "https://github.com/dlt-hub/dlt/blob/devel/dlt/load/load.py"
    click Normalize "https://github.com/dlt-hub/dlt/blob/devel/dlt/normalize/normalize.py"
    click Pipeline "https://github.com/dlt-hub/dlt/blob/devel/dlt/pipeline/pipeline.py"
    click Config "https://github.com/dlt-hub/dlt/blob/devel/dlt/common/configuration/__init__.py"
    click Schema "https://github.com/dlt-hub/dlt/blob/devel/dlt/common/schema/schema.py"
    click Sources "https://github.com/dlt-hub/dlt/blob/devel/dlt/sources/__init__.py"
    click Destinations "https://github.com/dlt-hub/dlt/blob/devel/dlt/destinations/adapters.py"
    click Helpers "https://github.com/dlt-hub/dlt/blob/devel/dlt/helpers/__init__.py"
    click CLI "https://github.com/dlt-hub/dlt/blob/devel/dlt/cli/_dlt.py"
    click Common "https://github.com/dlt-hub/dlt/blob/devel/dlt/common/__init__.py"
    click Writers "https://github.com/dlt-hub/dlt/blob/devel/dlt/common/data_writers/writers.py"
    click Types "https://github.com/dlt-hub/dlt/blob/devel/dlt/common/data_types/typing.py"
    click Storage "https://github.com/dlt-hub/dlt/blob/devel/dlt/common/storages/__init__.py"
    click State "https://github.com/dlt-hub/dlt/blob/devel/dlt/pipeline/state_sync.py"
    click SchemaEvo "https://github.com/dlt-hub/dlt/blob/devel/dlt/common/schema/migrations.py"
    click Incremental "https://github.com/dlt-hub/dlt/blob/devel/dlt/extract/incremental/__init__.py"
    click Monitor "https://github.com/dlt-hub/dlt/blob/devel/dlt/common/runtime/telemetry.py"

    %% Styling
    classDef core fill:#a8d1ff,stroke:#333,stroke-width:2px
    classDef common fill:#ffe6a8,stroke:#333,stroke-width:2px
    classDef integration fill:#d7a8ff,stroke:#333,stroke-width:2px
    
    class Extract,Normalize,Load,Pipeline core
    class Config,Schema,Storage,Types,Writers,SchemaEvo,State,Monitor common
    class CLI,Helpers,Sources,Destinations,Incremental integration
```
