# minhharry/grab-bootcamp-backend-2025 architecture

> Architecture diagram of the GitHub repository minhharry/grab-bootcamp-backend-2025, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/minhharry/grab-bootcamp-backend-2025
- Repository: https://github.com/minhharry/grab-bootcamp-backend-2025

## Overview

##### Project Type & Purpose
- This is a backend-only, data-centric FastAPI service that collects, processes, stores, and serves restaurant data (including image embeddings and recommendations).
- It combines a data-ingestion/ETL pipeline (Python, Spark, MinIO, Postgres), a vector-search component (Qdrant), and a REST API (FastAPI) exposing auth, restaurant, dish, recommendation, and image_search endpoints.

##### High-Level File Structure & Architectural Patterns
- **database/** – Docker-Compose orchestration, raw data collectors (Google Maps, Shopeefood), normalization & Spark-based ETL, data loaders for Postgres/MinIO, dummy data generation, Qdrant storage.
- **vector_db/** – Scripts to load or compute image embeddings into Qdrant.
- **src/** – FastAPI application organized by feature (routers), models (ORM), services, repositories, and global utils.
- **requirements/** – Requirements for CPU deployment.

Pattern:
- Data Pipeline (batch ELT) feeding Storage
- Micro-service style FastAPI with layered controllers → services → repositories
- Polyglot persistence: Postgres for relational, MinIO for blob, Qdrant for vector search

##### Main Components
A. Data Sources & Ingestion
- Google Maps & Shopeefood scrapers (Python CLI)
- (Optional) Kaggle notebook for image embeddings
B. Raw Data Storage
- Local filesystem or MinIO buckets for raw JSON
C. ETL & Processing
- Spark jobs & normalization scripts → transforms raw to canonical schemas
- UDFs, Spark configs, and local Python scripts
D. Processed Data Storage
- Postgres (restaurants, reviews, users, clicks)
- MinIO (images, blobs)
- Qdrant (image embeddings, vector index)
E. Backend API (FastAPI)
- Routers: auth, restaurant, dish, dummy, recommendation, image_search
- Models (SQLAlchemy), services, repositories
- Exception handlers, common schemas, utils
F. External Clients
- Web/mobile frontends or CLI consuming REST endpoints
- Adminer UI for DB inspection

##### Relationships & Data Flows
1. Scrapers → raw JSON files → MinIO / local FS
2. normalize_data.sh & Spark jobs → read raw → write processed → save to Postgres and/or MinIO
3. vector_db.load_embedding → read local CSV or compute via Kaggle → load into Qdrant
4. Backend API (FastAPI) on startup → connects to Postgres, MinIO (via MinIO SDK), Qdrant (via client)
5. Client requests → API controllers → services → repositories → underlying store (SQL, object, vector) → response

##### Important Technologies & Patterns
- FastAPI for HTTP layer (auto-generated OpenAPI docs)
- SQLAlchemy (or direct psycopg) for Postgres
- MinIO Python SDK for object storage
- PySpark for distributed data processing
- Qdrant vector DB for nearest-neighbor image search
- Docker Compose to orchestrate Postgres, MinIO, Qdrant locally
- Layered architecture: Router → Service → Repository → Database

##### Essential Details
- Distinguish between raw vs. processed data stores.
- How vector_db scripts feed Qdrant and how the image_search router interacts with it.
- Docker Compose network that ties Postgres, MinIO, and Qdrant for local dev.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Collection & ETL Zone
    subgraph "Zone 1: Data Collection & ETL"
        direction TB
        GGS["Google Maps Scraper"]:::pipeline
        SHO["Shopeefood Scraper"]:::pipeline
        RAW["Raw Data Storage (Local JSON)"]:::pipeline
        UPLOAD["Upload Raw Data to MinIO"]:::pipeline
        DOCKER["Docker-Compose Orchestration"]:::pipeline
        SPARK_ENTRY["Spark & Normalization Entry Point"]:::pipeline
        SPARK_CONF["Spark Configuration Files"]:::pipeline
        NORM_SCRIPT["Normalization Script"]:::pipeline
        PROC_MOD["Data Processing Modules"]:::pipeline
        UDFS["UDF Definitions"]:::pipeline
        SAVE_LOCAL["Save Processed Data to Local"]:::pipeline
        SAVE_MINIO["Save Processed Data to MinIO"]:::pipeline
        WRITE_PG["Write from Spark to Postgres"]:::pipeline
        INIT_SQL["Postgres Schema Initialization"]:::pipeline
        LOAD_SH["Batch Load Shell Script"]:::pipeline
        DUMMY_QDRANT["Dummy Qdrant Data & Storage"]:::pipeline

        GGS --> RAW
        SHO --> RAW
        RAW --> UPLOAD
        UPLOAD -->|"to MinIO"| DOCKER
        DOCKER --> SPARK_ENTRY
        SPARK_ENTRY --> SPARK_CONF
        SPARK_ENTRY --> NORM_SCRIPT
        NORM_SCRIPT --> PROC_MOD
        PROC_MOD --> UDFS
        PROC_MOD --> SAVE_LOCAL
        PROC_MOD --> SAVE_MINIO
        PROC_MOD --> WRITE_PG
        INIT_SQL --> LOAD_SH
        DUMMY_QDRANT -->|"seed Qdrant"| DOCKER
    end

    %% Vector DB Zone (part of pipeline)
    subgraph "Vector DB Scripts" 
        direction TB
        VEC_DB["Vector DB Scripts"]:::pipeline
        VEC_CONF["Qdrant Client Config"]:::pipeline
        VEC_EMB["Embedding Computation"]:::pipeline
        VEC_LOAD["Embedding Load CLI"]:::pipeline

        VEC_DB --> VEC_CONF
        VEC_DB --> VEC_EMB
        VEC_DB --> VEC_LOAD
        VEC_LOAD -->|"load embeddings"| QDRANT
    end

    %% Storage Systems Zone
    subgraph "Zone 2: Storage Systems"
        direction TB
        POSTGRES[(Postgres)]:::storage
        MINIO[(MinIO)]:::storage
        QDRANT[(Qdrant Vector DB)]:::storage
    end

    %% Backend Service Zone
    subgraph "Zone 3: Backend Service (FastAPI)"
        direction TB
        MAIN["FastAPI Backend Entry Point"]:::backend
        DB_CONF["Global App Config & DB Connections"]:::backend
        SCHEMAS["Common Schemas & Response Models"]:::backend
        EXC["Exception Handlers"]:::backend
        UTILS["Utility Functions"]:::backend

        subgraph "Routers (Controllers)"
            direction TB
            AUTH["Auth Router"]:::backend
            REST["Restaurant Router"]:::backend
            DISH["Dish Router"]:::backend
            REC["Recommendation Router"]:::backend
            IMG_SEARCH["Image Search Router"]:::backend
            DUMMY["Dummy Data Router"]:::backend
        end

        subgraph "Models (ORM Entities)"
            direction TB
            BASE["Base Models"]:::backend
            ENT_REST["Restaurant Model"]:::backend
            ENT_DISH["Dish Model"]:::backend
            ENT_REV["Review Model"]:::backend
            ENT_USER["User Model"]:::backend
            ENT_CLICK["User-Click Model"]:::backend
            ENT_IMG["Image Model"]:::backend
        end

        MAIN --> DB_CONF
        MAIN --> SCHEMAS
        MAIN --> EXC
        MAIN --> UTILS
        MAIN --> AUTH
        MAIN --> REST
        MAIN --> DISH
        MAIN --> REC
        MAIN --> IMG_SEARCH
        MAIN --> DUMMY
        AUTH --> BASE
        REST --> BASE
        DISH --> BASE
        REC --> BASE
        IMG_SEARCH --> BASE
        DUMMY --> BASE
    end

    %% External Clients Zone
    subgraph "Zone 4: External Clients & UIs"
        direction TB
        CLIENT["Web/Mobile Clients"]:::external
        ADMINER["Adminer UI"]:::external
    end

    %% Data Flows to Storage
    UPLOAD --> RT1((MinIO)):::storage
    WRITE_PG --> POSTGRES
    SAVE_MINIO --> MINIO
    SAVE_LOCAL --> RAW

    %% Backend connections to Storage
    MAIN -->|"connects to"| POSTGRES
    MAIN -->|"connects to"| MINIO
    MAIN -->|"connects to"| QDRANT

    %% Client interactions
    CLIENT -->|"REST API Requests"| MAIN
    ADMINER -->|"DB Inspection"| POSTGRES

    %% Embedding flows
    VEC_LOAD --> QDRANT

    %% Click Events
    click GGS "https://github.com/minhharry/grab-bootcamp-backend-2025/tree/main/database/collect_data/collect_data_ggmap"
    click SHO "https://github.com/minhharry/grab-bootcamp-backend-2025/tree/main/database/collect_data/collect_data_shopeefood"
    click RAW "https://github.com/minhharry/grab-bootcamp-backend-2025/tree/main/database/collect_data/raw_data"
    click UPLOAD "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/database/collect_data/upload_data_to_minIO.py"
    click DOCKER "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/database/docker-compose.yaml"
    click SPARK_ENTRY "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/database/normalize_data.sh"
    click SPARK_CONF "https://github.com/minhharry/grab-bootcamp-backend-2025/tree/main/database/spark-conf/"
    click NORM_SCRIPT "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/database/restaurant_data_processing/normalize_data.py"
    click PROC_MOD "https://github.com/minhharry/grab-bootcamp-backend-2025/tree/main/database/restaurant_data_processing"
    click SAVE_LOCAL "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/database/restaurant_data_processing/load_data/save_processed_data_to_local.py"
    click SAVE_MINIO "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/database/restaurant_data_processing/load_data/save_processed_data_to_minIO.py"
    click WRITE_PG "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/database/restaurant_data_processing/load_data/write_from_spark_to_postgres.py"
    click INIT_SQL "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/database/init.sql"
    click LOAD_SH "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/database/load_data.sh"
    click DUMMY_QDRANT "https://github.com/minhharry/grab-bootcamp-backend-2025/tree/main/database/dummy_users_data/qdrant_storage"
    click VEC_DB "https://github.com/minhharry/grab-bootcamp-backend-2025/tree/main/vector_db"
    click VEC_CONF "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/vector_db/database.py"
    click VEC_EMB "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/vector_db/embedding.py"
    click VEC_LOAD "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/vector_db/load_embedding.py"
    click MAIN "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/src/main.py"
    click DB_CONF "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/src/database.py"
    click SCHEMAS "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/src/common_schemas/response.py"
    click EXC "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/src/exception_handlers.py"
    click UTILS "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/src/utils.py"
    click AUTH "https://github.com/minhharry/grab-bootcamp-backend-2025/tree/main/src/routers/auth"
    click REST "https://github.com/minhharry/grab-bootcamp-backend-2025/tree/main/src/routers/restaurant"
    click DISH "https://github.com/minhharry/grab-bootcamp-backend-2025/tree/main/src/routers/dish"
    click REC "https://github.com/minhharry/grab-bootcamp-backend-2025/tree/main/src/routers/recommendation"
    click IMG_SEARCH "https://github.com/minhharry/grab-bootcamp-backend-2025/tree/main/src/routers/image_search"
    click DUMMY "https://github.com/minhharry/grab-bootcamp-backend-2025/tree/main/src/routers/dummy"
    click BASE "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/src/models/base.py"
    click ENT_REST "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/src/models/restaurant.py"
    click ENT_DISH "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/src/models/dish.py"
    click ENT_REV "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/src/models/review.py"
    click ENT_USER "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/src/models/user.py"
    click ENT_CLICK "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/src/models/user_restaurant_click.py"
    click ENT_IMG "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/src/models/image.py"
    click TESTS "https://github.com/minhharry/grab-bootcamp-backend-2025/tree/main/src/tests"
    click REQ_CPU "https://github.com/minhharry/grab-bootcamp-backend-2025/blob/main/requirements/requirements_cpu.txt"

    %% Styles
    classDef pipeline fill:#D0E6FF,stroke:#1E90FF
    classDef storage fill:#D0FFD6,stroke:#28A745
    classDef backend fill:#FFE5B4,stroke:#FF8C00
    classDef external fill:#EEEEEE,stroke:#999999
```
