How yi-lin-cheng/tjr101_group2 works
Project type and purpose
• This is an ETL‐oriented data‐engineering project: it ingests open government data, enriches it via Google Maps API and web scraping, then loads it into a MySQL database. Apache Airflow orchestrates the pipeline. GitHub Actions automates CI/CD, deploying to a GCP VM. A downstream LINE BOT reads the curated database to recommend itineraries.
File structure
• Top‐level orchestration & infra:
• .devcontainer/ (VSCode dev environment)
• Dockerfile (container for local development)
• .github/workflows/deploy.yml (CI/CD pipeline)
• Source code modules:
• src/accomo, src/accupass, src/food, src/klook, src/spot – each contains extract (e0x_), load (l0x_), transform (t0x_) scripts for that domain
• src/tasks – shared Airflow task logic (e.g., fuzzy matching, normalization)
• src/utils – common utilities (MySQL connection, encoding, notifications)
• Data directory: stores raw and intermediate CSVs per domain.
• pyproject.toml / poetry.lock: dependency management.
• README outlines objectives, tech stack, and CI/CD flow.
README for additional insights
• ETL objectives: automated extraction, enrichment, matching, and loading.
• Technologies: Python 3.12, Airflow, MySQL, Docker, GitHub Actions, GCP VM, pandas/Selenium/BeautifulSoup/rapidfuzz, Google Maps API.
• CI/CD: push to main → GitHub Actions → SSH to GCP VM → git pull → pipeline ready.
Essential details
– Show modular DAG structure (one per domain).
– Highlight shared utilities (tasks, utils packages).
– Capture external integrations (open data, Google Maps).
– Include CI/CD path from GitHub to VM.