How yusuffauziyan/scraping-google-maps works
Project Type and Purpose
• This is a backend‐focused, data-centric Python application that:
• Scrapes coffee shop data from Google Maps
• Cleans and stores that data
• Builds a TF-IDF + cosine-similarity recommendation engine
• Exposes results via a RESTful API
File Structure
• app/api: FastAPI (or Flask) routes for serving recommendations
• app/core: configuration (env variables, constants)
• app/scraping: module that hits Google Maps and writes raw JSON
• app/scripts: one-off ETL & model-building scripts (prepare_dataset.py, recommendation_coffee_shop.py, main.py)
• data/: raw JSON and cleaned CSV
• model/: pickled TF-IDF matrix and vectorizer
• docker-compose.yml: container orchestration
• postman_collection.json: sample API calls
Key Insights from README
• Workflow is linear: Scrape → Clean → Vectorize → Score → API
• External dependency: Google Maps data source
• Uses pandas, scikit-learn, numpy, requests
Main Components
a. External Service
• Google Maps API (HTTP/S requests for places)
b. Scraper Service
• coffee_scraper.py in app/scraping
• Reads config (app/core/config.py), writes data/coffe_raw.json
c. Data Processing (ETL)
• prepare_dataset.py cleans JSON → data/cleaned_coffee_data.csv
d. Recommendation Engine
• Scripts in app/scripts build TF-IDF (vectorizer.pkl) and similarity matrix (tfidf_matrix.pkl)
• Score calculation (rating_norm, reviews_norm)
e. API Layer
• app/api/routes.py loads pickles, accepts requests, returns JSON recommendations
f. Data & Model Storage
• data/ directory for raw & cleaned data
• model/ directory for pickled artifacts
g. Deployment
• Docker Compose orchestration (optional)
• .env for API keys/environment variables
Relationships & Data Flows
• Raw JSON → ETL → Cleaned CSV
• Cleaned CSV → Vectorizer → TF-IDF Matrix + Vectorizer Pickles
• Pickles → Recommendation Engine → Score calculation
• Pickles & Engine → API Layer → Client (e.g., Postman, Browser)
• Config (.env) applies to Scraper & API
Architectural Patterns & Principles
• Separation of Concerns: Scraper vs. ETL vs. Model vs. API
• Modularization: each script/module does one task
• Stateless API: loads model artifacts at startup, serves requests
• Data Pipeline: linear ETL + model preparation
Technologies & Frameworks
• Python 3.8+
• pandas, scikit-learn, numpy, requests
• FastAPI or Flask (in app/api)
• docker-compose for containerization