# nghiaai/weather_data_pipeline architecture

> Architecture diagram of the GitHub repository nghiaai/weather_data_pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nghiaai/weather_data_pipeline
- Repository: https://github.com/nghiaai/weather_data_pipeline

## Overview

##### Project Type & Purpose
- This project is a data pipeline application designed to automatically collect historical and current weather data from WeatherAPI. It uses Apache Airflow for scheduling and orchestrating tasks, and it stores raw data in MongoDB while structured data is persisted to PostgreSQL.
- The primary aim is to automate weather data collection, initially fetching 30 days of data and then switching to daily updates.

##### Analyzing the File Structure
- The top-level directories and files include:
- README.md – provides overall documentation.
- airflow/ – contains the Airflow-related components:
▫ dags/ – holds the DAG definition (e.g., main.py) that outlines the sequence of tasks.
▫ config/ – contains configuration files (config.py) where the API key and other settings are maintained.
- docker-compose.yaml – defines the container orchestration for multiple services (Airflow, MongoDB, PostgreSQL).
- dockerfile – used for building the Airflow container environment.
- The structure indicates clear separation of concerns: scheduling (Airflow), configuration, and container orchestration.

##### Key Components
a. Airflow:
- This as the central orchestrator.
- Subcomponents: the scheduler (for running DAGs), webserver (accessible at http://localhost:8080), and worker processes.
- Main DAG “weather_data_pipeline” containing tasks:
▫ check_mongo_connection,
▫ fetch_weather_data,
▫ add_data_to_postgres,
▫ check_data_postgres.
b. WeatherAPI:
c. MongoDB:
- Represented as the storage for raw weather data.
- That connectivity is verified through a specific task in the DAG.
d. PostgreSQL:
- Depicted as the database for structured weather data.
- That processed data from MongoDB is transferred here.
e. Docker & Docker Compose:
- That the entire system is containerized.
- That Docker Compose manages launching all necessary services (Airflow, MongoDB, PostgreSQL), as well as creating networking between them.

##### Relationships and Interactions
- Airflow DAG initiates connection checks and data fetching from WeatherAPI.
- Fetched data is stored in MongoDB.
- Another DAG task then transfers data from MongoDB to PostgreSQL.
- Dependencies such as:
- API calls to WeatherAPI.
- Data integrity and transformation steps between MongoDB and PostgreSQL.

##### Architectural Patterns and Technologies
- The project demonstrates a microservice/container-based approach, using Docker for service isolation.
- Airflow represents the workflow orchestration pattern.
- The pipeline can be viewed as adhering to ETL (Extract, Transform, Load) principles:
- Extraction from WeatherAPI.
- Loading raw data in MongoDB.
- Transformation and loading into PostgreSQL.
- Technologies to mention: Apache Airflow, MongoDB, PostgreSQL, Docker, Docker Compose, and WeatherAPI as the data source.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Containerization"
        DC1["Docker Compose"]:::container
        DC2["Dockerfile"]:::container
    end

    subgraph "Airflow Layer"
        A1["Airflow Scheduler"]:::orchestrator
        A2["Airflow Webserver"]:::orchestrator
        A3["Airflow Workers"]:::orchestrator
        subgraph "Weather Data Pipeline DAG"
            T1["check_mongo_connection"]:::task
            T2["fetch_weather_data"]:::task
            T3["add_data_to_postgres"]:::task
            T4["check_data_postgres"]:::task
        end
    end

    WAPI["WeatherAPI"]:::external
    MongoDB["MongoDB"]:::database
    Postgres["PostgreSQL"]:::database

    %% Containerization launches services
    DC1 --> A1
    DC1 --> MongoDB
    DC1 --> Postgres

    %% Airflow internal flow
    A1 --> A2
    A2 --> A3
    T1 --> T2
    T2 --> T3
    T3 --> T4

    %% Pipeline interactions
    T2 -->|"fetches"| WAPI
    WAPI -->|"stores"| MongoDB
    T1 -->|"verifies"| MongoDB
    T3 -->|"transfers"| Postgres
    T4 -->|"verifies"| Postgres

    %% Click Events
    click DC1 "https://github.com/nghiaai/weather_data_pipeline/blob/main/docker-compose.yaml"
    click DC2 "https://github.com/nghiaai/weather_data_pipeline/tree/main/dockerfile"
    click A1 "https://github.com/nghiaai/weather_data_pipeline/blob/main/airflow/dags/main.py"
    click A1 "https://github.com/nghiaai/weather_data_pipeline/blob/main/airflow/config/config.py"
    click WAPI "https://github.com/nghiaai/weather_data_pipeline/blob/main/airflow/config/config.py"
    click MongoDB "https://github.com/nghiaai/weather_data_pipeline/blob/main/docker-compose.yaml"
    click Postgres "https://github.com/nghiaai/weather_data_pipeline/blob/main/docker-compose.yaml"

    %% Styles
    classDef container fill:#f9e79f,stroke:#7d6608,stroke-width:2px
    classDef orchestrator fill:#aed6f1,stroke:#1b4f72,stroke-width:2px
    classDef task fill:#fcf3cf,stroke:#7d6608,stroke-width:2px
    classDef external fill:#d5f5e3,stroke:#1e8449,stroke-width:2px
    classDef database fill:#f9c4c4,stroke:#922b21,stroke-width:2px
```
