# yusuffauziyan/scraping-google-maps architecture

> Architecture diagram of the GitHub repository yusuffauziyan/scraping-google-maps, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yusuffauziyan/scraping-google-maps
- Repository: https://github.com/yusuffauziyan/scraping-google-maps

## Overview

##### Project Type and Purpose
- This is a backend‐focused, data-centric Python application that:
- Scrapes coffee shop data from Google Maps
- Cleans and stores that data
- Builds a TF-IDF + cosine-similarity recommendation engine
- Exposes results via a RESTful API

##### File Structure
- app/api: FastAPI (or Flask) routes for serving recommendations
- app/core: configuration (env variables, constants)
- app/scraping: module that hits Google Maps and writes raw JSON
- app/scripts: one-off ETL & model-building scripts (prepare_dataset.py, recommendation_coffee_shop.py, main.py)
- data/: raw JSON and cleaned CSV
- model/: pickled TF-IDF matrix and vectorizer
- docker-compose.yml: container orchestration
- postman_collection.json: sample API calls

##### Key Insights from README
- Workflow is linear: Scrape → Clean → Vectorize → Score → API
- External dependency: Google Maps data source
- Uses pandas, scikit-learn, numpy, requests

##### Main Components
a. External Service
- Google Maps API (HTTP/S requests for places)
b. Scraper Service
- coffee_scraper.py in app/scraping
- Reads config (app/core/config.py), writes data/coffe_raw.json
c. Data Processing (ETL)
- prepare_dataset.py cleans JSON → data/cleaned_coffee_data.csv
d. Recommendation Engine
- Scripts in app/scripts build TF-IDF (vectorizer.pkl) and similarity matrix (tfidf_matrix.pkl)
- Score calculation (rating_norm, reviews_norm)
e. API Layer
- app/api/routes.py loads pickles, accepts requests, returns JSON recommendations
f. Data & Model Storage
- data/ directory for raw & cleaned data
- model/ directory for pickled artifacts
g. Deployment
- Docker Compose orchestration (optional)
- .env for API keys/environment variables

##### Relationships & Data Flows
- Raw JSON → ETL → Cleaned CSV
- Cleaned CSV → Vectorizer → TF-IDF Matrix + Vectorizer Pickles
- Pickles → Recommendation Engine → Score calculation
- Pickles & Engine → API Layer → Client (e.g., Postman, Browser)
- Config (.env) applies to Scraper & API

##### Architectural Patterns & Principles
- Separation of Concerns: Scraper vs. ETL vs. Model vs. API
- Modularization: each script/module does one task
- Stateless API: loads model artifacts at startup, serves requests
- Data Pipeline: linear ETL + model preparation

##### Technologies & Frameworks
- Python 3.8+
- pandas, scikit-learn, numpy, requests
- FastAPI or Flask (in app/api)
- docker-compose for containerization

## Diagram (Mermaid)

```mermaid
flowchart TD
    direction TB

    subgraph "Docker Compose Orchestration" 
        direction TB
        DOCKER["docker-compose.yml"]:::infra
        subgraph "Services"
            direction TB
            SCRAPER["Scraper Service\ncoffee_scraper.py"]:::service
            ETL["Data Processing (ETL)\nprepare_dataset.py"]:::service
            SCORE["Model Trainer\nrecomendation_coffe_shop.py"]:::service
            API_ROUTES["API Routes\nroutes.py"]:::service
            API_MAIN["API Server Entry Point\nmain.py"]:::service
        end
    end

    GM["Google Maps API"]:::external
    CONFIG["Configuration\nconfig.py"]:::config
    ENV_EX["Env Example\n.env.example"]:::config

    RAW["Raw Data\ncoffe_raw.json"]:::storage
    CLEAN["Cleaned Data\ncleaned_coffee_data.csv"]:::storage
    VECT["Vectorizer Artifact\nvectorizer.pkl"]:::storage
    MATRIX["TF-IDF Matrix\ntfidf_matrix.pkl"]:::storage

    POSTMAN["Postman Client"]:::client

    %% Data Flow
    GM -->|"HTTP/JSON"| SCRAPER
    CONFIG -->|env| SCRAPER
    ENV_EX -->|sample| CONFIG

    SCRAPER -->|"writes JSON"| RAW
    RAW -->|"input JSON"| ETL
    ETL -->|"writes CSV"| CLEAN

    CLEAN -->|"input CSV"| SCORE
    SCORE -->|"produces pickle"| VECT
    SCORE -->|"produces pickle"| MATRIX

    VECT -->|"loads pickles"| API_MAIN
    MATRIX -->|"loads pickles"| API_MAIN
    CONFIG -->|env| API_MAIN
    API_ROUTES -->|registers routes| API_MAIN
    API_MAIN -->|"HTTP/JSON"| POSTMAN

    %% Click Events
    click SCRAPER "https://github.com/yusuffauziyan/scraping-google-maps/blob/main/app/scraping/coffee_scraper.py"
    click CONFIG "https://github.com/yusuffauziyan/scraping-google-maps/blob/main/app/core/config.py"
    click ETL "https://github.com/yusuffauziyan/scraping-google-maps/blob/main/app/scripts/prepare_dataset.py"
    click SCORE "https://github.com/yusuffauziyan/scraping-google-maps/blob/main/app/scripts/recomendation_coffe_shop.py"
    click API_ROUTES "https://github.com/yusuffauziyan/scraping-google-maps/blob/main/app/api/routes.py"
    click API_MAIN "https://github.com/yusuffauziyan/scraping-google-maps/blob/main/app/main.py"
    click RAW "https://github.com/yusuffauziyan/scraping-google-maps/blob/main/data/coffe_raw.json"
    click CLEAN "https://github.com/yusuffauziyan/scraping-google-maps/blob/main/data/cleaned_coffee_data.csv"
    click VECT "https://github.com/yusuffauziyan/scraping-google-maps/blob/main/model/vectorizer.pkl"
    click MATRIX "https://github.com/yusuffauziyan/scraping-google-maps/blob/main/model/tfidf_matrix.pkl"
    click DOCKER "https://github.com/yusuffauziyan/scraping-google-maps/blob/main/docker-compose.yml"
    click ENV_EX "https://github.com/yusuffauziyan/scraping-google-maps/blob/main/.env.example"
    click POSTMAN "https://github.com/yusuffauziyan/scraping-google-maps/blob/main/postman_collection.json"

    %% Styles
    classDef external fill:#f9c74f,stroke:#333,stroke-width:1px;
    classDef service fill:#90be6d,stroke:#333,stroke-width:1px;
    classDef storage fill:#43aa8b,stroke:#333,stroke-width:1px;
    classDef config fill:#f8961e,stroke:#333,stroke-width:1px;
    classDef client fill:#577590,stroke:#333,stroke-width:1px;
    classDef infra fill:#f94144,stroke:#333,stroke-width:1px;
```
