# yi-lin-cheng/tjr101_group2 architecture

> Architecture diagram of the GitHub repository yi-lin-cheng/tjr101_group2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yi-lin-cheng/tjr101_group2
- Repository: https://github.com/yi-lin-cheng/tjr101_group2

## Overview

##### Project type and purpose
- This is an ETL‐oriented data‐engineering project: it ingests open government data, enriches it via Google Maps API and web scraping, then loads it into a MySQL database. Apache Airflow orchestrates the pipeline. GitHub Actions automates CI/CD, deploying to a GCP VM. A downstream LINE BOT reads the curated database to recommend itineraries.

##### File structure
- Top‐level orchestration & infra:
- .devcontainer/ (VSCode dev environment)
- Dockerfile (container for local development)
- .github/workflows/deploy.yml (CI/CD pipeline)
- Source code modules:
- src/accomo, src/accupass, src/food, src/klook, src/spot – each contains extract (e0x_), load (l0x_), transform (t0x_) scripts for that domain
- src/tasks – shared Airflow task logic (e.g., fuzzy matching, normalization)
- src/utils – common utilities (MySQL connection, encoding, notifications)
- Data directory: stores raw and intermediate CSVs per domain.
- pyproject.toml / poetry.lock: dependency management.
- README outlines objectives, tech stack, and CI/CD flow.

##### README for additional insights
- ETL objectives: automated extraction, enrichment, matching, and loading.
- Technologies: Python 3.12, Airflow, MySQL, Docker, GitHub Actions, GCP VM, pandas/Selenium/BeautifulSoup/rapidfuzz, Google Maps API.
- CI/CD: push to main → GitHub Actions → SSH to GCP VM → git pull → pipeline ready.

##### Essential details
– Show modular DAG structure (one per domain).
– Highlight shared utilities (tasks, utils packages).
– Capture external integrations (open data, Google Maps).
– Include CI/CD path from GitHub to VM.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Code & CI/CD"
        DevEnv[".devcontainer/"]:::code
        Dockerfile["Dockerfile"]:::code
        DeployYML[".github/workflows/deploy.yml"]:::cicd
        PyProj["pyproject.toml"]:::code
        Poetry["poetry.lock"]:::code
    end

    DevEnv -->|"configured by"| Dockerfile
    Dockerfile -->|"build container"| DevPush["Push to GitHub"]:::code
    DevPush -->|"trigger"| CI["GitHub Actions"]:::cicd
    CI -->|"deploy via SSH"| GCPVM["GCP VM"]:::infra

    subgraph "Airflow on GCP VM"
        Scheduler["Scheduler"]:::airflow
        WebUI["Webserver/UI"]:::airflow
        Workers["Workers"]:::airflow

        Scheduler -->|"schedules"| DAGs["DAG Definitions"]:::airflow
        WebUI -->|"monitoring"| Scheduler

        subgraph "Domain DAGs"
            AccomoDAG["accomo DAG"]:::airflow
            AccupassDAG["accupass DAG"]:::airflow
            FoodDAG["food DAG"]:::airflow
            KlookDAG["klook DAG"]:::airflow
            SpotDAG["spot DAG"]:::airflow
        end

        DAGs --> AccomoDAG
        DAGs --> AccupassDAG
        DAGs --> FoodDAG
        DAGs --> KlookDAG
        DAGs --> SpotDAG

        AccomoDAG --> Workers
        AccupassDAG --> Workers
        FoodDAG --> Workers
        KlookDAG --> Workers
        SpotDAG --> Workers
    end

    subgraph "ETL Modules"
        AccomoMod["src/accomo/"]:::code
        AccupassMod["src/accupass/"]:::code
        FoodMod["src/food/"]:::code
        KlookMod["src/klook/"]:::code
        SpotMod["src/spot/"]:::code

        TasksMod["src/tasks/"]:::code
        UtilsMod["src/utils/"]:::code
    end

    AccomoDAG -->|"executes"| AccomoMod
    AccupassDAG -->|"executes"| AccupassMod
    FoodDAG -->|"executes"| FoodMod
    KlookDAG -->|"executes"| KlookMod
    SpotDAG -->|"executes"| SpotMod

    Workers --> TasksMod
    Workers --> UtilsMod

    subgraph "Data Flow"
        GovData["Government Open Data APIs"]:::external
        GoogleMaps["Google Maps API"]:::external
        CSVStore["data/<domain>/ (raw & intermediate CSVs)"]:::datastore
        MySQLDB["MySQL Database"]:::datastore
        LineBot["LINE BOT"]:::bot
    end

    GovData -->|"extract"| AccomoMod
    GovData -->|"extract"| AccupassMod
    GovData -->|"extract"| FoodMod
    GovData -->|"extract"| KlookMod
    GovData -->|"extract"| SpotMod

    GoogleMaps -->|"enrich"| FoodMod
    GoogleMaps -->|"enrich"| SpotMod
    GoogleMaps -->|"enrich"| AccomoMod

    AccomoMod -->|"write CSV"| CSVStore
    AccupassMod -->|"write CSV"| CSVStore
    FoodMod -->|"write CSV"| CSVStore
    KlookMod -->|"write CSV"| CSVStore
    SpotMod -->|"write CSV"| CSVStore

    TasksMod -->|"transform"| AccomoMod
    TasksMod -->|"transform"| AccupassMod
    TasksMod -->|"transform"| FoodMod
    TasksMod -->|"transform"| KlookMod
    TasksMod -->|"transform"| SpotMod

    UtilsMod -->|"load to"| MySQLDB

    LineBot -->|"query"| MySQLDB

    %% Click Events
    click DevEnv "https://github.com/yi-lin-cheng/tjr101_group2/blob/main/.devcontainer/devcontainer.json"
    click DevEnv "https://github.com/yi-lin-cheng/tjr101_group2/blob/main/.devcontainer/postCreateCommand.sh"
    click Dockerfile "https://github.com/yi-lin-cheng/tjr101_group2/tree/main/Dockerfile"
    click DeployYML "https://github.com/yi-lin-cheng/tjr101_group2/blob/main/.github/workflows/deploy.yml"
    click PyProj "https://github.com/yi-lin-cheng/tjr101_group2/blob/main/pyproject.toml"
    click Poetry "https://github.com/yi-lin-cheng/tjr101_group2/blob/main/poetry.lock"
    click AccomoMod "https://github.com/yi-lin-cheng/tjr101_group2/tree/main/src/accomo/"
    click AccupassMod "https://github.com/yi-lin-cheng/tjr101_group2/tree/main/src/accupass/"
    click FoodMod "https://github.com/yi-lin-cheng/tjr101_group2/tree/main/src/food/"
    click KlookMod "https://github.com/yi-lin-cheng/tjr101_group2/tree/main/src/klook/"
    click SpotMod "https://github.com/yi-lin-cheng/tjr101_group2/tree/main/src/spot/"
    click TasksMod "https://github.com/yi-lin-cheng/tjr101_group2/tree/main/src/tasks/"
    click UtilsMod "https://github.com/yi-lin-cheng/tjr101_group2/tree/main/src/utils/"

    classDef code fill:#D0E6FF,stroke:#0056b3,color:#0056b3;
    classDef cicd fill:#FFF0D0,stroke:#b36e00,color:#b36e00;
    classDef infra fill:#E8FFE8,stroke:#00a000,color:#007a00;
    classDef airflow fill:#FFF8D0,stroke:#b39e00,color:#b39e00;
    classDef datastore fill:#F0FFF0,stroke:#009900,color:#006600;
    classDef external fill:#F0F0FF,stroke:#0000b3,color:#000099;
    classDef bot fill:#E8FFF8,stroke:#00a080,color:#008060;
```
