# tmph2003/doantotnghiep architecture

> Architecture diagram of the GitHub repository tmph2003/doantotnghiep, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tmph2003/doantotnghiep
- Repository: https://github.com/tmph2003/doantotnghiep

## Overview

##### Project Type and Core Purpose
- This is a full-fledged data warehouse solution that uses Airflow to orchestrate ETL pipelines. The solution uses several containerized services (Airflow, Postgres, Redis [as mentioned in the README], Trino, ClickHouse, Superset, Hive Metastore, etc.) to manage workflow orchestration, metadata, ETL processing, analytical querying, and data visualization.
- That there is also a “warehouse” component implemented in custom ETL Python modules (in the plugins/warehouse folder) that encapsulate data extraction, DDL management, and transformation logic.

##### File Structure for Architectural Insights
- Top-Level Folders:
- containers: Contains the Dockerfiles and configurations for all major services such as Airflow, Postgres, ClickHouse, Superset, Hive-Metastore, and Trino.
- dags: Contains Airflow DAG definitions, including ETL workflows and data orchestration.
- plugins: Offers custom Python modules for data transformation and schema (ETL) logic that Airflow tasks execute.
- Assets (like CSV files) that may be imported into DAGs.
- data: Contains persistent storage/data files for the services, notably the Postgres database and potentially MinIO (indicated by .minio sys subfolders).
- The Docker Compose YAML files (e.g., docker-compose-bi.yaml, docker-compose-compute.yaml) illustrate the orchestration of these services.

##### Component Identification
a. Container Orchestrator Environment
- Docker and Docker Compose (the foundational layer for running containers).
- Each containerized service:
- Airflow: further broken down into its three core services – webserver, scheduler, and worker – as well as auxiliary services (e.g., airflow-init, triggered, log grooming).
- Postgres: Used primarily for Airflow metadata (and possibly as a component in the warehouse).
- Redis: Acts as the message broker for Airflow scheduler (as mentioned in the README; although not explicitly seen in the file tree, it is an integral element).
- Trino: A distributed SQL query engine for querying data across multiple sources.
- ClickHouse: Likely used as an analytical database.
- Superset: For data visualization and dashboarding.
- Hive-Metastore: Supporting metadata for data processing.
b. Data Processing and ETL:
- Airflow DAGs located in the “dags” directory that orchestrate task execution.
- Custom ETL modules in the “plugins/warehouse/etl” directory, including separate logic for “bronze” (raw/extraction) and “silver” (transformation/cleaning) layers.
c. Data Repositories:
- Persistent “data” folders used for storing Postgres data and other files (e.g., MinIO for a data lake structure).
- External systems (if any) that feed into or extract from the warehouse (for trading data input) are also represented.

##### Relationships and Interactions
- How Airflow—as the orchestrator—kicks off ETL tasks (invoking modules in “plugins”) which load/extract data from the “include” and “data” storage areas.
- How Airflow interacts with the Postgres container (for metadata) and the Redis broker.
- How transformed data might then be queried by Trino or loaded into ClickHouse for analytics.
- How Superset connects to these databases (e.g., ClickHouse/Trino) for visualization.
- Bidirectional flows if needed (for example, logging data from Airflow flows back to persistent storage).

##### Architectural Patterns and Design Principles
- A microservices or containerized architecture: each service is self-contained but interacts through well-defined interfaces.
- Separation of concerns: workflow orchestration (Airflow), data storage (Postgres, ClickHouse), query engine (Trino), and presentation (Superset).
- Extensibility points: custom ETL logic in plugins and modular DAG definitions enable flexible future enhancements.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Infrastructure Layer
    subgraph "Infrastructure"
        DC1["Docker Compose: bi"]:::container
        DC2["Docker Compose: compute"]:::container
    end

    %% ETL Orchestration Layer
    subgraph "ETL Orchestration"
        AF["Airflow Service"]:::orchestration
        AFWeb["Airflow Webserver"]:::orchestration
        AFScheduler["Airflow Scheduler"]:::orchestration
        AFWorker["Airflow Worker"]:::orchestration
        RED["Redis Broker"]:::orchestration
    end

    %% Data Processing Layer
    subgraph "Data Processing"
        DAG["Airflow DAGs"]:::etl
        ETL["Custom ETL Modules"]:::etl
        COMMON["Common Library"]:::etl
    end

    %% Data Storage & Query Layer
    subgraph "Data Storage & Query"
        PG["Postgres DB"]:::storage
        TR["Trino Query Engine"]:::storage
        CH["ClickHouse Analytics DB"]:::storage
        HM["Hive Metastore"]:::storage
    end

    %% Data Repositories
    subgraph "Data Repositories"
        ASSETS["Include Assets"]:::storage
        DATA["Persistent Data Storage"]:::storage
    end

    %% Visualization Layer
    subgraph "Visualization"
        SS["Superset Dashboard"]:::visualization
    end

    %% Relationships and Data Flows
    %% Infrastructure triggers services
    DC1 -->|"orchestrates"| AF
    DC2 -->|"orchestrates"| AF
    DC1 -->|"orchestrates"| PG
    DC1 -->|"orchestrates"| TR
    DC1 -->|"orchestrates"| CH
    DC1 -->|"orchestrates"| SS
    DC1 -->|"orchestrates"| HM

    %% Airflow internal components
    AF --> AFWeb
    AF --> AFScheduler
    AF --> AFWorker

    %% Airflow dependencies
    AFScheduler -->|"metadata"| PG
    AFScheduler -->|"broker"| RED

    %% ETL workflow
    AFWorker -->|"launches"| DAG
    AF -->|"triggers"| DAG
    DAG -->|"executes"| ETL
    COMMON -.-> ETL
    ETL -->|"loads/transforms"| DATA
    ASSETS -->|"provides data"| DAG

    %% Query and Visualization connections
    TR -->|"serves queries to"| SS
    CH -->|"serves analytics to"| SS

    %% Click Events
    click DC1 "https://github.com/tmph2003/doantotnghiep/blob/dev/docker-compose-bi.yaml"
    click DC2 "https://github.com/tmph2003/doantotnghiep/blob/dev/docker-compose-compute.yaml"
    click AF "https://github.com/tmph2003/doantotnghiep/tree/dev/containers/airflow/"
    click DAG "https://github.com/tmph2003/doantotnghiep/tree/dev/dags/warehouse/"
    click PG "https://github.com/tmph2003/doantotnghiep/tree/dev/containers/postgres/"
    click PG "https://github.com/tmph2003/doantotnghiep/tree/dev/data/postgres-data/"
    click TR "https://github.com/tmph2003/doantotnghiep/tree/dev/containers/trino/"
    click CH "https://github.com/tmph2003/doantotnghiep/tree/dev/containers/clickhouse/"
    click SS "https://github.com/tmph2003/doantotnghiep/tree/dev/containers/superset/"
    click HM "https://github.com/tmph2003/doantotnghiep/tree/dev/containers/hive-metastore/"
    click ETL "https://github.com/tmph2003/doantotnghiep/tree/dev/plugins/warehouse/etl/"
    click COMMON "https://github.com/tmph2003/doantotnghiep/tree/dev/plugins/warehouse/common/"
    click ASSETS "https://github.com/tmph2003/doantotnghiep/tree/dev/include/csv/"
    click DATA "https://github.com/tmph2003/doantotnghiep/tree/dev/data/"

    %% Styles
    classDef container fill:#ffd699,stroke:#333,stroke-width:2px;
    classDef orchestration fill:#ffb3b3,stroke:#333,stroke-width:2px;
    classDef storage fill:#cce5ff,stroke:#333,stroke-width:2px;
    classDef etl fill:#d4edda,stroke:#333,stroke-width:2px;
    classDef visualization fill:#e6ccff,stroke:#333,stroke-width:2px;
```
