How ayalaadaili/etl_airflow works
Project Type and Purpose
• This is a data-engineering pipeline and dashboarding “full-stack” project. It uses Apache Airflow (Python) to orchestrate an ETL pipeline that reads raw data, transforms it, and loads it into a SQLite store, while using Dash and Streamlit for interactive visualization. All components are containerized via Docker Compose, with PostgreSQL as Airflow’s metadata database.
File Structure Highlights
• Top-level:
• Dockerfile, Dockerfile.streamlit – images for services
• docker-compose.yaml – orchestrates Airflow, Postgres, SQLite volume, Dash, Streamlit
• app.py, dashboard.py – entrypoints for Dash and Streamlit apps
• dags/etl.py – the pipeline definition (DAG)
• Data/data.rar – raw data source
• logs/ – Airflow task logs
• requirements.txt – Python dependencies
• Patterns:
• “dags” folder—Airflow’s DAG definitions (Extraction, Transformation, Load)
• Separate scripts for Dash (dashboard.py) and Streamlit (app.py)
• Docker Compose network linking all services
Key Architecture Insights from README
• Components:
• Apache Airflow (Scheduler, Webserver, Workers)
• PostgreSQL (Airflow metadata DB)
• SQLite (transformed data store)
• Dash (http://localhost:8050) for interactive charts
• Streamlit (http://localhost:8501) for exploratory UI
• Deployment:
• Single Docker Compose stack, services defined in docker-compose.yaml
• Ports exposed for Airflow UI (8080), Dash, and Streamlit
• Workflow:
• Raw data extracted from local archive (Data/)
• Airflow tasks orchestrate ETL steps defined in etl.py
• Cleaned data loaded into SQLite
• Dash and Streamlit read from SQLite for visualization