# thuantd2207/web-news-main architecture

> Architecture diagram of the GitHub repository thuantd2207/web-news-main, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thuantd2207/web-news-main
- Repository: https://github.com/thuantd2207/web-news-main

## Overview

##### Main Components
- Airflow Environment – This includes the scheduler, DAGs (in the “dags” directory), and task logs (found in the “logs” folders). Airflow orchestrates tasks such as scraping data (using the crawler scripts), processing it (using functions like database_func.py and model_func.py), and coordinating data flows.
- Flask Web Application – Located in the “web-app” folder (with app.py and index.html). This component provides the user interface which displays the news data.
- PostgreSQL Database – Even though a dedicated folder is not present, the README mentions PostgreSQL as the storage backend for news data.
- Docker / Docker-Compose Setup – With Dockerfile and docker-compose.yaml, the entire system infrastructure (Airflow, Flask, PostgreSQL) is containerized, ensuring consistent deployment.
- Environment & Configuration – The existence of a .env file and other configuration files help define environment variables and settings, which you might represent as a configuration or settings layer.

##### Relationships and Interactions
- Web UI: Illustrate that the Flask app queries the PostgreSQL database to fetch the latest news data and displays it to users.
- Containerization: Include the Docker container environment as an encapsulating layer around the Airflow, Flask, and PostgreSQL components. You could show these containers connected via a Docker network.
- Logging: Optionally include a note about the logging subsystem where Airflow writes out logs (via log directories) for monitoring and troubleshooting.

##### Architectural Patterns and Design Principles
- Modularity – The separation between components (Airflow for orchestration and scraping, Flask for presentation, and PostgreSQL for persistence) demonstrates a modular design.
- Containerization – Use of Docker demonstrates microservice principles and ensures each service can be developed, managed, and deployed independently.
- Workflow Orchestration – The use of Airflow indicates the project follows a workflow management pattern, with clear separation between tasks defined in individual DAG files (crawler_dag.py, etc.).
- Extensibility – The directory structure with clear segregation (logs, DAGs, web-app) helps in scaling and adding new functionality.

## Diagram (Mermaid)

```mermaid
flowchart TD
    ENV[".env"]:::config
    subgraph "Deployment"
        dockerCompose["docker-compose.yaml"]:::containerConf
        dockerfile["Dockerfile"]:::containerConf
    end
    ENV -->|"loads"| dockerCompose

    %% Containerization arrows from Deployment to services
    dockerCompose -->|"containerizes"| air
    dockerCompose -->|"containerizes"| flask
    dockerCompose -->|"containerizes"| db

    subgraph "Docker Environment"
        subgraph "Data Pipeline Layer"
            air["Airflow Environment"]:::orchestration
            air -->|"schedules"| dag1["Crawler DAG"]:::orchestration
            air -->|"schedules"| dag2["Crawler Functions"]:::orchestration
            air -->|"schedules"| dag3["Database Functions"]:::orchestration
            air -->|"schedules"| dag4["Model Functions"]:::orchestration
            dag1 -->|"logs"| logs["Task Logs"]:::logging
            dag2 -->|"logs"| logs
            dag3 -->|"logs"| logs
            dag4 -->|"logs"| logs
            air -->|"DataIngestion"| db
        end
        subgraph "Presentation Layer"
            flask["Flask Web Application"]:::frontend
            index["Frontend Template"]:::frontend
            flask -->|"renders"| index
            flask -->|"DataRetrieval"| db
            db["PostgreSQL Database"]:::database
        end
    end

    click ENV "https://github.com/thuantd2207/web-news-main/blob/main/.env"
    click dockerCompose "https://github.com/thuantd2207/web-news-main/blob/main/docker-compose.yaml"
    click dockerfile "https://github.com/thuantd2207/web-news-main/tree/main/Dockerfile"
    click dag1 "https://github.com/thuantd2207/web-news-main/blob/main/dags/crawler_dag.py"
    click dag2 "https://github.com/thuantd2207/web-news-main/blob/main/dags/crawler_func.py"
    click dag3 "https://github.com/thuantd2207/web-news-main/blob/main/dags/database_func.py"
    click dag4 "https://github.com/thuantd2207/web-news-main/blob/main/dags/model_func.py"
    click logs "https://github.com/thuantd2207/web-news-main/tree/main/logs/"
    click flask "https://github.com/thuantd2207/web-news-main/blob/main/web-app/app.py"
    click index "https://github.com/thuantd2207/web-news-main/blob/main/web-app/templates/index.html"

    classDef orchestration fill:#C6E5FF,stroke:#039BE5,stroke-width:2px;
    classDef logging fill:#FFF3E0,stroke:#FB8C00,stroke-width:2px;
    classDef database fill:#E8F5E9,stroke:#43A047,stroke-width:2px;
    classDef frontend fill:#F3E5F5,stroke:#8E24AA,stroke-width:2px;
    classDef containerConf fill:#FFEBEE,stroke:#E53935,stroke-width:2px;
    classDef config fill:#E0F7FA,stroke:#00ACC1,stroke-width:2px;
```
