How nghiaai/weather_data_pipeline works
Project Type & Purpose
• This project is a data pipeline application designed to automatically collect historical and current weather data from WeatherAPI. It uses Apache Airflow for scheduling and orchestrating tasks, and it stores raw data in MongoDB while structured data is persisted to PostgreSQL.
• The primary aim is to automate weather data collection, initially fetching 30 days of data and then switching to daily updates.
Analyzing the File Structure
• The top-level directories and files include:
• README.md – provides overall documentation.
• airflow/ – contains the Airflow-related components:
▫ dags/ – holds the DAG definition (e.g., main.py) that outlines the sequence of tasks.
▫ config/ – contains configuration files (config.py) where the API key and other settings are maintained.
• docker-compose.yaml – defines the container orchestration for multiple services (Airflow, MongoDB, PostgreSQL).
• dockerfile – used for building the Airflow container environment.
• The structure indicates clear separation of concerns: scheduling (Airflow), configuration, and container orchestration.
Key Components
a. Airflow:
• This as the central orchestrator.
• Subcomponents: the scheduler (for running DAGs), webserver (accessible at http://localhost:8080), and worker processes.
• Main DAG “weather_data_pipeline” containing tasks:
▫ check_mongo_connection,
▫ fetch_weather_data,
▫ add_data_to_postgres,
▫ check_data_postgres.
b. WeatherAPI:
c. MongoDB:
• Represented as the storage for raw weather data.
• That connectivity is verified through a specific task in the DAG.
d. PostgreSQL:
• Depicted as the database for structured weather data.
• That processed data from MongoDB is transferred here.
e. Docker & Docker Compose:
• That the entire system is containerized.
• That Docker Compose manages launching all necessary services (Airflow, MongoDB, PostgreSQL), as well as creating networking between them.
Relationships and Interactions
• Airflow DAG initiates connection checks and data fetching from WeatherAPI.
• Fetched data is stored in MongoDB.
• Another DAG task then transfers data from MongoDB to PostgreSQL.
• Dependencies such as:
• API calls to WeatherAPI.
• Data integrity and transformation steps between MongoDB and PostgreSQL.
Architectural Patterns and Technologies
• The project demonstrates a microservice/container-based approach, using Docker for service isolation.
• Airflow represents the workflow orchestration pattern.
• The pipeline can be viewed as adhering to ETL (Extract, Transform, Load) principles:
• Extraction from WeatherAPI.
• Loading raw data in MongoDB.
• Transformation and loading into PostgreSQL.
• Technologies to mention: Apache Airflow, MongoDB, PostgreSQL, Docker, Docker Compose, and WeatherAPI as the data source.