How dhruvrajsinh21/cricket_match_scraper works
Project Type and Purpose
• This is a backend-focused, real-time data-pipeline system for monitoring live cricket matches. It uses web scraping (Playwright) to ingest data, stores it in MongoDB, schedules and triggers jobs via APScheduler, and pushes updates to clients over WebSockets.
File Structure Analysis
• Top-level modules:
• scrapers/ → Plays the role of data ingestion from an external site.
• scheduler/ → Orchestrates periodic jobs and real-time triggers.
• database/ → Encapsulates MongoDB connection and CRUD operations.
• websock/ → WebSocket server and notification logic.
• main.py → Bootstrap entry point tying components together.
• requirements.txt → Declares Python dependencies (Playwright, APScheduler, motor/pymongo, websockets).
README Insights
• Confirms use of:
• Playwright-based scrapers (match list + match detail).
• APScheduler for “monitor.py” (periodic scraping) and “trigger_jobs.py” (live-match triggers).
• MongoDB for semi-structured match data.
• WebSocket server on ws://localhost:6789 delivering update events.
Main Components
a. External Systems
• Cricket website (e.g. https://crex.live)
• MongoDB instance (localhost:27017)
• WebSocket clients (browsers, scripts)
b. Internal Services
1. Scrapers
• match_list_scraper.py
• match_detail_scraper.py
2. Scheduler
• monitor.py
• trigger_jobs.py
3. Database Layer
• db_connection.py
• db_queries.py
4. WebSocket Layer
• server.py
• notify.py
5. Orchestration
• main.py
Component Interactions and Data Flow
• Monitor → every X minutes → invokes match_list_scraper → fetches schedules → writes to MongoDB.
• Trigger_Jobs → polls upcoming/live matches → invokes match_detail_scraper → updates live data in MongoDB → emits “match_live” events to notify.py.
• WebSocket server listens for events from notify.py → pushes JSON payloads to connected clients.
• Clients subscribe over ws://localhost:6789 → receive real-time update messages.
Architectural Patterns and Design Principles
• Modular separation by concern: scraping, persistence, scheduling, and notification.
• Event-driven flow for real-time updates.
• Layered design: ingestion → persistence → notification.
• Use of job scheduler (APScheduler) ensures decoupled timed workflows.
• Asynchronous WebSocket server for low-latency push.
Detailed Elements to Capture
• Entry point: main.py spawns Scheduler and WebSocket server threads.
• monitor.py versus trigger_jobs.py roles.
• db_connection setup and query functions.
• clients subscribing channel and payload schema (match_id, event, score, timestamp).
• External service (Playwright) invocation.