How minhharry/grab-bootcamp-backend-2025 works
Project Type & Purpose
• This is a backend-only, data-centric FastAPI service that collects, processes, stores, and serves restaurant data (including image embeddings and recommendations).
• It combines a data-ingestion/ETL pipeline (Python, Spark, MinIO, Postgres), a vector-search component (Qdrant), and a REST API (FastAPI) exposing auth, restaurant, dish, recommendation, and image_search endpoints.
High-Level File Structure & Architectural Patterns
• database/ – Docker-Compose orchestration, raw data collectors (Google Maps, Shopeefood), normalization & Spark-based ETL, data loaders for Postgres/MinIO, dummy data generation, Qdrant storage.
• vector_db/ – Scripts to load or compute image embeddings into Qdrant.
• src/ – FastAPI application organized by feature (routers), models (ORM), services, repositories, and global utils.
• requirements/ – Requirements for CPU deployment.
Pattern:
• Data Pipeline (batch ELT) feeding Storage
• Micro-service style FastAPI with layered controllers → services → repositories
• Polyglot persistence: Postgres for relational, MinIO for blob, Qdrant for vector search
Main Components
A. Data Sources & Ingestion
• Google Maps & Shopeefood scrapers (Python CLI)
• (Optional) Kaggle notebook for image embeddings
B. Raw Data Storage
• Local filesystem or MinIO buckets for raw JSON
C. ETL & Processing
• Spark jobs & normalization scripts → transforms raw to canonical schemas
• UDFs, Spark configs, and local Python scripts
D. Processed Data Storage
• Postgres (restaurants, reviews, users, clicks)
• MinIO (images, blobs)
• Qdrant (image embeddings, vector index)
E. Backend API (FastAPI)
• Routers: auth, restaurant, dish, dummy, recommendation, image_search
• Models (SQLAlchemy), services, repositories
• Exception handlers, common schemas, utils
F. External Clients
• Web/mobile frontends or CLI consuming REST endpoints
• Adminer UI for DB inspection
Relationships & Data Flows
1. Scrapers → raw JSON files → MinIO / local FS
2. normalize_data.sh & Spark jobs → read raw → write processed → save to Postgres and/or MinIO
3. vector_db.load_embedding → read local CSV or compute via Kaggle → load into Qdrant
4. Backend API (FastAPI) on startup → connects to Postgres, MinIO (via MinIO SDK), Qdrant (via client)
5. Client requests → API controllers → services → repositories → underlying store (SQL, object, vector) → response
Important Technologies & Patterns
• FastAPI for HTTP layer (auto-generated OpenAPI docs)
• SQLAlchemy (or direct psycopg) for Postgres
• MinIO Python SDK for object storage
• PySpark for distributed data processing
• Qdrant vector DB for nearest-neighbor image search
• Docker Compose to orchestrate Postgres, MinIO, Qdrant locally
• Layered architecture: Router → Service → Repository → Database
Essential Details
• Distinguish between raw vs. processed data stores.
• How vector_db scripts feed Qdrant and how the image_search router interacts with it.
• Docker Compose network that ties Postgres, MinIO, and Qdrant for local dev.