# piyushsoni27/bms_movies_data_extracter architecture

> Architecture diagram of the GitHub repository piyushsoni27/bms_movies_data_extracter, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/piyushsoni27/bms_movies_data_extracter
- Repository: https://github.com/piyushsoni27/bms_movies_data_extracter

## Overview

##### Project Type and Purpose
- This is an open-source Python-based ETL (Extract, Transform, Load) tool that scrapes unstructured show, movie, theatre, and city data from the BookMyShow website and pushes it into a remote MySQL server.
- It uses Selenium (via chromedriver) for data extraction, Python classes for domain modeling and cleaning, and an SQL module for persistence.

##### Key Components (derived from file tree & README)
- Scraping Layer
– bms_scrapper.py
- Orchestrates the overall scrape for a region (e.g. get_topten_movies, get_nearby_theatres).
– Supporting scripts: bms_now_showing.py, bms_rating.py, bms_reviews.py
- Domain Model / Transformation Layer
– city_class.py (CityTable)
– movie_class.py (MovieTable)
– theatre_show.py (TheatreTable + ShowTable)
- Persistence Layer
– sql.py (connects and writes to remote MySQL)
- Artifacts / Outputs
– Local CSV files: CITY_Table.csv, MOVIE_Table.csv, THEATRE_Table.csv, SHOW_Table.csv
- External Dependencies
– chromedriver_win32/chromedriver.exe (Selenium driver)
– Remote MySQL server

##### Relationships and Data Flow
1. The “Scraping Layer” scripts invoke Selenium + chromedriver to fetch raw HTML.
2. Raw HTML is handed off to the “Domain Model” classes which parse, clean, and map data into row objects for cities, movies, theatres, shows.
3. Each class writes interim results to its corresponding CSV (optional) for offline inspection.
4. sql.py reads the cleaned data (or receives it via class methods) and issues INSERT/UPDATE statements to the remote MySQL server.

##### Architectural Patterns and Principles
- Pipeline pattern: clear Extract → Transform → Load stages.
- Separation of concerns: distinct modules for scraping, data modeling, and persistence.
- Modularity and extensibility: easy to add new scrappers (ratings, reviews), new data tables, or swap out the SQL backend.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Scraping Layer"
        BMS["bms_scrapper.py"]:::scrape
        NOW["bms_now_showing.py"]:::scrape
        RAT["bms_rating.py"]:::scrape
        REV["bms_reviews.py"]:::scrape
    end

    EXT["Selenium + ChromeDriver"]:::external

    subgraph "Domain Model / Transformation Layer"
        CITY["city_class.py → CITY_Table"]:::trans
        MOV["movie_class.py → MOVIE_Table"]:::trans
        THEA["theatre_show.py → THEATRE_Table & SHOW_Table"]:::trans
    end

    subgraph "Artifacts / Outputs"
        C1["CITY_Table.csv"]:::artifact
        C2["MOVIE_Table.csv"]:::artifact
        C3["THEATRE_Table.csv"]:::artifact
        C4["SHOW_Table.csv"]:::artifact
    end

    SQLM["sql.py"]:::persistence
    DB["Remote MySQL Server"]:::db

    BMS -->|"uses"| EXT
    NOW -->|"uses"| EXT
    RAT -->|"uses"| EXT
    REV -->|"uses"| EXT

    EXT -->|"raw HTML"| CITY
    EXT -->|"raw HTML"| MOV
    EXT -->|"raw HTML"| THEA

    CITY -->|"parsed rows"| C1
    CITY -->|"clean data"| SQLM
    MOV -->|"parsed rows"| C2
    MOV -->|"clean data"| SQLM
    THEA -->|"parsed rows"| C3
    THEA -->|"parsed rows"| C4
    THEA -->|"clean data"| SQLM

    SQLM -->|"INSERT/UPDATE"| DB

    classDef scrape fill:#ADD8E6,stroke:#000
    classDef trans fill:#90EE90,stroke:#000
    classDef persistence fill:#FFA500,stroke:#000
    classDef external fill:#D3D3D3,stroke:#000
    classDef artifact fill:#F0E68C,stroke:#000
    classDef db fill:#D3D3D3,stroke:#000,stroke-width:2px

    click BMS "https://github.com/piyushsoni27/bms_movies_data_extracter/blob/master/bms_scrapper.py"
    click NOW "https://github.com/piyushsoni27/bms_movies_data_extracter/blob/master/bms_now_showing.py"
    click RAT "https://github.com/piyushsoni27/bms_movies_data_extracter/blob/master/bms_rating.py"
    click REV "https://github.com/piyushsoni27/bms_movies_data_extracter/blob/master/bms_reviews.py"
    click CITY "https://github.com/piyushsoni27/bms_movies_data_extracter/blob/master/city_class.py"
    click MOV "https://github.com/piyushsoni27/bms_movies_data_extracter/blob/master/movie_class.py"
    click THEA "https://github.com/piyushsoni27/bms_movies_data_extracter/blob/master/theatre_show.py"
    click C1 "https://github.com/piyushsoni27/bms_movies_data_extracter/blob/master/CITY_Table.csv"
    click C2 "https://github.com/piyushsoni27/bms_movies_data_extracter/blob/master/MOVIE_Table.csv"
    click C3 "https://github.com/piyushsoni27/bms_movies_data_extracter/blob/master/THEATRE_Table.csv"
    click C4 "https://github.com/piyushsoni27/bms_movies_data_extracter/blob/master/SHOW_Table.csv"
    click SQLM "https://github.com/piyushsoni27/bms_movies_data_extracter/blob/master/sql.py"
    click EXT "https://github.com/piyushsoni27/bms_movies_data_extracter/blob/master/chromedriver_win32/chromedriver.exe"
```
