# abhisheksingh1180/fakerapi-elt-pipeline architecture

> Architecture diagram of the GitHub repository abhisheksingh1180/fakerapi-elt-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abhisheksingh1180/fakerapi-elt-pipeline
- Repository: https://github.com/abhisheksingh1180/fakerapi-elt-pipeline

## Overview

##### Project type and purpose
- This is a data‐engineering ELT pipeline project that uses Apache Airflow to orchestrate the movement and transformation of synthetic “product” data (from FakerAPI) into Snowflake via Google Cloud Storage (GCS).
- Purpose: Extract product data from an HTTP API, stage it in GCS, load into Snowflake tables (stage → refined → confirmed), then run Snowflake stored procedures to apply business logic.

##### File structure
- Top level
- README.md – project overview
- docker-compose.yaml, dockerfile – containerize Airflow environment with local executor.
- dags/product_pipeline.py – Airflow DAG definition (tasks: extract, load, transform).
- packages/fakerAPI.py – Python module to call the Faker API and write CSV.
- sql/ddl/* – Snowflake DDL scripts to create raw, refined, confirmed schemas and views.
- sql/scripts/* – SQL to move data between stages inside Snowflake.
- temp/*.csv – sample extract output.

##### README for additional insights
- Confirms key components: Airflow, Faker API, GCS staging, Snowflake stored procedures.
- Describes workflow: extract → stage → load → transform, all orchestrated by Airflow.

a. Main components
1. FakerAPI module (Python) – HTTP client that produces product CSVs
2. Apache Airflow – scheduler & executor running in Docker containers
3. Google Cloud Storage (GCS) – staging bucket for raw CSVs
4. Snowflake – cloud data warehouse hosting stage, refined, confirmed tables + stored procedures
5. Local filesystem or temp folder – intermediate storage in container
6. Docker network – connects Airflow services to GCS and Snowflake
b. Relationships & interactions
- The Airflow DAG’s first task calls FakerAPI.py → writes CSV to temp/ or directly to GCS.
- Next Airflow tasks upload CSVs to GCS buckets.
- Subsequent Airflow tasks issue Snowflake COPY INTO commands to load staging tables.
- Final DAG tasks invoke Snowflake stored procedures (via SnowflakeHook) to transform data in place.
- Airflow’s webserver and scheduler monitor & log each task.
c. Architectural patterns & design principles
- ELT pattern: extract raw, load raw, then transform in-database.
- Containerized microservices: Airflow components (scheduler, webserver, worker) run in Docker for isolation & reproducibility.
- Modular code: separate packages/fakerAPI and DAG definitions.
- Infrastructure as code: Docker and Docker Compose for environment consistency.
d. Technologies & frameworks
- Apache Airflow (Python)
- Docker / Docker Compose
- Google Cloud Storage (GCS)
- Snowflake (SQL DDL + stored procedures)
- FakerAPI (Python library)

##### Detailed but not over‐engineered
- How Airflow’s DAG steps map to GCS uploads and Snowflake loads.
- Orchestration flow; show monitoring (Airflow WebUI) if helpful.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Container Orchestration
    subgraph "Container Orchestration"
        DC["Docker Compose"]:::infra
        DF["Dockerfile"]:::infra
    end

    %% Airflow Container
    subgraph "Docker Network"
        subgraph "Airflow Container" 
            A_WEB["Airflow Webserver"]:::container
            A_SCH["Airflow Scheduler"]:::container
            A_WRK["Airflow Worker"]:::container
            DAG["Airflow DAG: product_pipeline.py"]:::python
        end
    end

    %% Code Modules
    FAG["FakerAPI Module"]:::python

    %% Temporary Storage
    TEMP["Temp CSV Storage"]:::infra

    %% Cloud Storage
    GCS["GCS Bucket: raw-data"]:::gcs

    %% Snowflake Data Warehouse
    subgraph "Snowflake Data Warehouse"
        subgraph "Schemas & Tables"
            STG["Stage Tables"]:::snowflake
            REF["Refined Tables"]:::snowflake
            CON["Confirmed Tables"]:::snowflake
        end
        DDL["Snowflake DDL Scripts"]:::snowflake
        SCRIPTS["Snowflake ELT Load Scripts"]:::snowflake
        COLDEF["Common Column Definitions"]:::snowflake
    end

    %% Documentation
    ARCH["Architecture Diagram"]:::infra

    %% Data Flow
    DAG -->|"calls FakerAPI"| FAG
    FAG -->|"writes CSV"| TEMP
    DAG -->|"uploads CSV"| GCS
    TEMP -->|"optional local read"| DAG
    GCS -->|"COPY INTO Stage"| STG
    STG -->|"transform via SP"| REF
    REF -->|"transform via SP"| CON

    %% Orchestration Links
    DAG -->|"managed by"| A_SCH
    DAG -->|"logs to"| A_WEB
    A_SCH --> A_WRK

    %% Scripts & DDL usage
    DAG -->|"executes DDL"| DDL
    DAG -->|"executes ELT scripts"| SCRIPTS
    DDL --> COLDEF

    %% Documentation Link
    DC --> DAG
    ARCH

    %% Click Events
    click FAG "https://github.com/abhisheksingh1180/fakerapi-elt-pipeline/blob/main/packages/fakerAPI.py"
    click DAG "https://github.com/abhisheksingh1180/fakerapi-elt-pipeline/blob/main/dags/product_pipeline.py"
    click DC "https://github.com/abhisheksingh1180/fakerapi-elt-pipeline/blob/main/docker-compose.yaml"
    click DF "https://github.com/abhisheksingh1180/fakerapi-elt-pipeline/tree/main/dockerfile"
    click DDL "https://github.com/abhisheksingh1180/fakerapi-elt-pipeline/tree/main/sql/ddl/"
    click SCRIPTS "https://github.com/abhisheksingh1180/fakerapi-elt-pipeline/tree/main/sql/scripts/"
    click TEMP "https://github.com/abhisheksingh1180/fakerapi-elt-pipeline/tree/main/temp/"
    click ARCH "https://github.com/abhisheksingh1180/fakerapi-elt-pipeline/blob/main/diagram/etl_pipeline.png"
    click COLDEF "https://github.com/abhisheksingh1180/fakerapi-elt-pipeline/blob/main/sql/product_coloumn_def.txt"

    %% Styles
    classDef python fill:#ffcc00,stroke:#333,stroke-width:1px
    classDef container fill:#99ccff,stroke:#333,stroke-width:1px
    classDef gcs fill:#ccffcc,stroke:#333,stroke-width:1px
    classDef snowflake fill:#ccccff,stroke:#333,stroke-width:1px
    classDef infra fill:#ffcccc,stroke:#333,stroke-width:1px
```
