# bruin-data/ingestr architecture

> Architecture diagram of the GitHub repository bruin-data/ingestr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bruin-data/ingestr
- Repository: https://github.com/bruin-data/ingestr

## Overview

##### Main Components

a) Source Layer:
- Sources into categories:
- Databases (PostgreSQL, BigQuery, Snowflake, etc.)
- Platforms/APIs (Adjust, Airtable, Facebook Ads, etc.)
- File Systems (CSV, S3)

b) Core Processing Layer:
- Command Line Interface (main.py)
- Source Handler (sources.py)
- Destination Handler (destinations.py)
- Factory Component (factory.py)
- Table Definition Manager (table_definition.py)
- Time Management (time.py)
- Telemetry System (telemetry/)

c) Destination Layer:
- Supported Databases (PostgreSQL, BigQuery, Snowflake, etc.)

##### Component Relationships

**[Source Layer] → [Core Processing Layer] → [Destination Layer]**

With these specific interactions:
- CLI receives user input (URIs and parameters)
- Factory creates appropriate source/destination handlers
- Source handlers extract data
- Core processing transforms and prepares data
- Destination handlers load data

##### Key Design Elements

- Rectangles: Core components
- Hexagons: Processing modules

- Blue: Source components
- Green: Core processing
- Orange: Destination components
- Gray: Supporting systems (telemetry, time management)

##### Implementation Details to Highlight

a) Source Handling:
**[Source URI] → [Source Factory] → [Specific Source Handler] → [Data Extraction]**

b) Data Flow:
**[Extracted Data] → [Processing/Transformation] → [Loading Strategy (append/merge/delete+insert)] → [Destination]**

##### Architectural Patterns to Emphasize

- Factory Pattern (for source/destination creation)
- Strategy Pattern (for different loading strategies)
- Adapter Pattern (for different source/destination implementations)

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles
    classDef source fill:#a8d1f0,stroke:#333
    classDef core fill:#b7e3c0,stroke:#333
    classDef destination fill:#ffd4a8,stroke:#333
    classDef support fill:#e0e0e0,stroke:#333
    classDef database fill:#f9f,stroke:#333
    classDef api fill:#ff9,stroke:#333
    classDef file fill:#ddd,stroke:#333

    %% Source Layer
    subgraph Sources
        DB_Sources["Database Sources"]
        API_Sources["API/Platform Sources"]
        File_Sources["File System Sources"]

        subgraph Databases
            MongoDB["MongoDB"]:::database
            Kafka["Kafka"]:::database
        end

        subgraph APIs
            FB["Facebook Ads"]:::api
            GA["Google Analytics"]:::api
            Shopify["Shopify"]:::api
            Stripe["Stripe Analytics"]:::api
        end

        subgraph Files
            FS["File Systems"]:::file
        end
    end

    %% Core Processing Layer
    subgraph Core
        CLI["Command Line Interface"]:::core
        Factory["Factory Component"]:::core
        SrcHandler["Source Handler"]:::core
        DestHandler["Destination Handler"]:::core
        TableDef["Table Definition Manager"]:::core
        TimeManager["Time Management"]:::core
        Filter["Filter System"]:::core
        Version["Version Management"]:::core
    end

    %% Support Systems
    subgraph Support
        Telemetry["Telemetry System"]:::support
        TestData["Test Data"]:::support
    end

    %% Destination Layer
    subgraph Destinations
        SQL_DB["SQL Database Handler"]:::destination
    end

    %% Relationships
    DB_Sources --> Factory
    API_Sources --> Factory
    File_Sources --> Factory
    Factory --> SrcHandler
    Factory --> DestHandler
    CLI --> Factory
    SrcHandler --> TableDef
    DestHandler --> TableDef
    TimeManager --> SrcHandler
    Filter --> SrcHandler
    Version --> Core
    SrcHandler --> SQL_DB
    Telemetry --> Core

    %% Click Events
    click CLI "https://github.com/bruin-data/ingestr/blob/main/ingestr/main.py"
    click SrcHandler "https://github.com/bruin-data/ingestr/blob/main/ingestr/src/sources.py"
    click DestHandler "https://github.com/bruin-data/ingestr/blob/main/ingestr/src/destinations.py"
    click Factory "https://github.com/bruin-data/ingestr/blob/main/ingestr/src/factory.py"
    click TableDef "https://github.com/bruin-data/ingestr/blob/main/ingestr/src/table_definition.py"
    click TimeManager "https://github.com/bruin-data/ingestr/blob/main/ingestr/src/time.py"
    click Telemetry "https://github.com/bruin-data/ingestr/tree/main/ingestr/src/telemetry/"
    click SQL_DB "https://github.com/bruin-data/ingestr/tree/main/ingestr/src/sql_database/"
    click FS "https://github.com/bruin-data/ingestr/tree/main/ingestr/src/filesystem/"
    click FB "https://github.com/bruin-data/ingestr/tree/main/ingestr/src/facebook_ads/"
    click GA "https://github.com/bruin-data/ingestr/tree/main/ingestr/src/google_analytics/"
    click Shopify "https://github.com/bruin-data/ingestr/tree/main/ingestr/src/shopify/"
    click Stripe "https://github.com/bruin-data/ingestr/tree/main/ingestr/src/stripe_analytics/"
    click MongoDB "https://github.com/bruin-data/ingestr/tree/main/ingestr/src/mongodb/"
    click Kafka "https://github.com/bruin-data/ingestr/tree/main/ingestr/src/kafka/"
    click Version "https://github.com/bruin-data/ingestr/blob/main/ingestr/src/version.py"
    click Filter "https://github.com/bruin-data/ingestr/blob/main/ingestr/src/filters.py"
    click TestData "https://github.com/bruin-data/ingestr/tree/main/ingestr/testdata/"
```
