How dhivyahari16/reddit_eli5_etl works
Project Type and Purpose
• This is a full‐stack data pipeline project that uses Apache Airflow to orchestrate daily ETL jobs pulling data from Reddit (r/explainlikeimfive), storing raw files in object storage (MinIO), loading and transforming them in PostgreSQL, and visualizing KPIs in Metabase.
File Structure Analysis
• Top level:
• .env, Dockerfile, docker-compose.yaml, requirements.txt → containerization & configuration
• README.md → project overview & setup
• dags/ → Airflow DAG definitions and Python task source
– data_pipeline.py → DAG orchestration
– python_src/ → individual ETL task scripts (scraping, staging, loading, aggregation)
Key Components
a. Orchestration Layer
• Apache Airflow (Scheduler, Webserver, Workers) running the data_pipeline DAG
b. Ingestion Layer
• Reddit Scraper (Python script using PRAW)
• MinIO Object Storage for raw CSV blobs
c. Transformation & Storage
• PostgreSQL database with schemas/tables: raw_reddit_posts, posts, comments, authors, kpi_aggregates
d. Visualization Layer
• Metabase reading from PostgreSQL to render dashboards
e. Deployment & Infrastructure
• Docker Compose network connecting Airflow, MinIO, Postgres, Metabase
• Environment variables (.env) or Airflow Connections
Component Relationships & Data Flows
1) Airflow DAG triggers on schedule →
2) Task “Reddit_scrapper.py” calls PRAW API → writes CSVs to MinIO →
3) Task “Minio_upload.py” ensures staging in object storage →
4) Task “Insert_to_landing_zone.py” extracts from MinIO → loads raw_reddit_posts →
5) Task “Insert_to_DWH.py” transforms and loads posts, comments, authors →
6) Task “Aggregate_insert.py” computes daily KPIs into kpi_aggregates →
7) Metabase queries kpi_aggregates to display dashboards
Architectural Patterns & Principles
• Modular ETL tasks following single‐responsibility
• Batch processing orchestrated by a DAG
• Separation of raw (landing), staging, and production tables
• Infrastructure as code via Docker Compose
• Externalized configuration using .env / Airflow Connections
Technologies & Frameworks
• Orchestration: Apache Airflow
• Ingestion: PRAW (Reddit API), Python
• Object Storage: MinIO
• Database: PostgreSQL
• Visualization: Metabase
• Packaging & Deployment: Docker, Docker Compose