# dhivyahari16/reddit_eli5_etl architecture

> Architecture diagram of the GitHub repository dhivyahari16/reddit_eli5_etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dhivyahari16/reddit_eli5_etl
- Repository: https://github.com/dhivyahari16/reddit_eli5_etl

## Overview

##### Project Type and Purpose
- This is a full‐stack data pipeline project that uses Apache Airflow to orchestrate daily ETL jobs pulling data from Reddit (r/explainlikeimfive), storing raw files in object storage (MinIO), loading and transforming them in PostgreSQL, and visualizing KPIs in Metabase.

##### File Structure Analysis
- Top level:
- .env, Dockerfile, docker-compose.yaml, requirements.txt → containerization & configuration
- README.md → project overview & setup
- dags/ → Airflow DAG definitions and Python task source
– data_pipeline.py → DAG orchestration
– python_src/ → individual ETL task scripts (scraping, staging, loading, aggregation)

##### Key Components
a. Orchestration Layer
- Apache Airflow (Scheduler, Webserver, Workers) running the data_pipeline DAG
b. Ingestion Layer
- Reddit Scraper (Python script using PRAW)
- MinIO Object Storage for raw CSV blobs
c. Transformation & Storage
- PostgreSQL database with schemas/tables: raw_reddit_posts, posts, comments, authors, kpi_aggregates
d. Visualization Layer
- Metabase reading from PostgreSQL to render dashboards
e. Deployment & Infrastructure
- Docker Compose network connecting Airflow, MinIO, Postgres, Metabase
- Environment variables (.env) or Airflow Connections

##### Component Relationships & Data Flows
1) Airflow DAG triggers on schedule →
2) Task “Reddit_scrapper.py” calls PRAW API → writes CSVs to MinIO →
3) Task “Minio_upload.py” ensures staging in object storage →
4) Task “Insert_to_landing_zone.py” extracts from MinIO → loads raw_reddit_posts →
5) Task “Insert_to_DWH.py” transforms and loads posts, comments, authors →
6) Task “Aggregate_insert.py” computes daily KPIs into kpi_aggregates →
7) Metabase queries kpi_aggregates to display dashboards

##### Architectural Patterns & Principles
- Modular ETL tasks following single‐responsibility
- Batch processing orchestrated by a DAG
- Separation of raw (landing), staging, and production tables
- Infrastructure as code via Docker Compose
- Externalized configuration using .env / Airflow Connections

##### Technologies & Frameworks
- Orchestration: Apache Airflow
- Ingestion: PRAW (Reddit API), Python
- Object Storage: MinIO
- Database: PostgreSQL
- Visualization: Metabase
- Packaging & Deployment: Docker, Docker Compose

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External API
    RedditAPI["Reddit API"]:::external

    %% Configuration & Docs
    ENV[".env"]:::config
    DOCKERFILE["Dockerfile"]:::config
    COMPOSE["docker-compose.yaml"]:::config
    REQ["requirements.txt"]:::config
    README["README.md"]:::config

    %% Docker Network Boundary
    subgraph "Docker Compose Network"
        direction TB
        AF["Apache Airflow<br/>(Scheduler, Webserver, Workers)"]:::orchestration
        MinIO["MinIO Object Storage"]:::storage
        Postgres["PostgreSQL Database"]:::storage
        Metabase["Metabase"]:::visualization
    end

    %% DAG Definition
    DAG["data_pipeline.py"]:::orchestration
    AF -->|"triggers daily"| DAG

    %% Ingestion Tasks
    RS["Reddit_scrapper.py"]:::ingestion
    MU["Minio_upload.py"]:::ingestion
    IL["Insert_to_landing_zone.py"]:::ingestion
    DWH["Insert_to_DWH.py"]:::ingestion
    AG["Aggregate_insert.py"]:::ingestion

    %% Data Artifact
    CSV["Raw CSV"]:::artifact

    %% Flow: DAG to tasks
    DAG -->|"calls PRAW, scrapes posts"| RS
    RS -->|"writes CSV"| CSV
    CSV -->|"stores blob"| MinIO
    MinIO -->|"ensures staging"| MU
    MU -->|"objects staged"| MinIO
    MinIO -->|"extract CSV"| IL
    IL -->|"SQL INSERT raw_reddit_posts"| Postgres
    Postgres -->|"read landing data"| DWH
    DWH -->|"transform & load posts, comments, authors"| Postgres
    Postgres -->|"read transformed data"| AG
    AG -->|"compute daily KPIs into kpi_aggregates"| Postgres
    Metabase -->|"queries kpi_aggregates"| Postgres

    %% Config flows
    ENV --> AF
    ENV --> RS
    ENV --> MU
    ENV --> IL
    ENV --> DWH
    ENV --> AG

    %% Click Events
    click DAG "https://github.com/dhivyahari16/reddit_eli5_etl/blob/main/dags/data_pipeline.py"
    click RS "https://github.com/dhivyahari16/reddit_eli5_etl/blob/main/dags/python_src/Reddit_scrapper.py"
    click MU "https://github.com/dhivyahari16/reddit_eli5_etl/blob/main/dags/python_src/Minio_upload.py"
    click IL "https://github.com/dhivyahari16/reddit_eli5_etl/blob/main/dags/python_src/Insert_to_landing_zone.py"
    click DWH "https://github.com/dhivyahari16/reddit_eli5_etl/blob/main/dags/python_src/Insert_to_DWH.py"
    click AG "https://github.com/dhivyahari16/reddit_eli5_etl/blob/main/dags/python_src/Aggregate_insert.py"
    click COMPOSE "https://github.com/dhivyahari16/reddit_eli5_etl/blob/main/docker-compose.yaml"
    click DOCKERFILE "https://github.com/dhivyahari16/reddit_eli5_etl/tree/main/Dockerfile"
    click ENV "https://github.com/dhivyahari16/reddit_eli5_etl/blob/main/.env"
    click REQ "https://github.com/dhivyahari16/reddit_eli5_etl/blob/main/requirements.txt"
    click README "https://github.com/dhivyahari16/reddit_eli5_etl/blob/main/README.md"

    %% Styling
    classDef orchestration fill:#cce5ff,stroke:#004085,stroke-width:1px;
    classDef ingestion fill:#d4edda,stroke:#155724,stroke-width:1px;
    classDef storage fill:#fff3cd,stroke:#856404,stroke-width:1px;
    classDef visualization fill:#d6d8f9,stroke:#383d7c,stroke-width:1px;
    classDef artifact fill:#e2e3e5,stroke:#6c757d,stroke-width:1px;
    classDef external fill:#f8d7da,stroke:#721c24,stroke-width:1px;
    classDef config fill:#f1f1f1,stroke:#333,stroke-width:1px;
```
