How piyushsoni27/bms_movies_data_extracter works
Project Type and Purpose
• This is an open-source Python-based ETL (Extract, Transform, Load) tool that scrapes unstructured show, movie, theatre, and city data from the BookMyShow website and pushes it into a remote MySQL server.
• It uses Selenium (via chromedriver) for data extraction, Python classes for domain modeling and cleaning, and an SQL module for persistence.
Key Components (derived from file tree & README)
• Scraping Layer
– bms_scrapper.py
• Orchestrates the overall scrape for a region (e.g. get_topten_movies, get_nearby_theatres).
– Supporting scripts: bms_now_showing.py, bms_rating.py, bms_reviews.py
• Domain Model / Transformation Layer
– city_class.py (CityTable)
– movie_class.py (MovieTable)
– theatre_show.py (TheatreTable + ShowTable)
• Persistence Layer
– sql.py (connects and writes to remote MySQL)
• Artifacts / Outputs
– Local CSV files: CITY_Table.csv, MOVIE_Table.csv, THEATRE_Table.csv, SHOW_Table.csv
• External Dependencies
– chromedriver_win32/chromedriver.exe (Selenium driver)
– Remote MySQL server
Relationships and Data Flow
1. The “Scraping Layer” scripts invoke Selenium + chromedriver to fetch raw HTML.
2. Raw HTML is handed off to the “Domain Model” classes which parse, clean, and map data into row objects for cities, movies, theatres, shows.
3. Each class writes interim results to its corresponding CSV (optional) for offline inspection.
4. sql.py reads the cleaned data (or receives it via class methods) and issues INSERT/UPDATE statements to the remote MySQL server.
Architectural Patterns and Principles
• Pipeline pattern: clear Extract → Transform → Load stages.
• Separation of concerns: distinct modules for scraping, data modeling, and persistence.
• Modularity and extensibility: easy to add new scrappers (ratings, reviews), new data tables, or swap out the SQL backend.