# dhruvrajsinh21/cricket_match_scraper architecture

> Architecture diagram of the GitHub repository dhruvrajsinh21/cricket_match_scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dhruvrajsinh21/cricket_match_scraper
- Repository: https://github.com/dhruvrajsinh21/cricket_match_scraper

## Overview

##### Project Type and Purpose
- This is a backend-focused, real-time data-pipeline system for monitoring live cricket matches. It uses web scraping (Playwright) to ingest data, stores it in MongoDB, schedules and triggers jobs via APScheduler, and pushes updates to clients over WebSockets.

##### File Structure Analysis
- Top-level modules:
- scrapers/           → Plays the role of data ingestion from an external site.
- scheduler/          → Orchestrates periodic jobs and real-time triggers.
- database/           → Encapsulates MongoDB connection and CRUD operations.
- websock/            → WebSocket server and notification logic.
- main.py             → Bootstrap entry point tying components together.
- requirements.txt    → Declares Python dependencies (Playwright, APScheduler, motor/pymongo, websockets).

##### README Insights
- Confirms use of:
- Playwright-based scrapers (match list + match detail).
- APScheduler for “monitor.py” (periodic scraping) and “trigger_jobs.py” (live-match triggers).
- MongoDB for semi-structured match data.
- WebSocket server on ws://localhost:6789 delivering update events.

##### Main Components
a. External Systems
- Cricket website (e.g. https://crex.live)
- MongoDB instance (localhost:27017)
- WebSocket clients (browsers, scripts)
b. Internal Services
1. Scrapers
- match_list_scraper.py
- match_detail_scraper.py
2. Scheduler
- monitor.py
- trigger_jobs.py
3. Database Layer
- db_connection.py
- db_queries.py
4. WebSocket Layer
- server.py
- notify.py
5. Orchestration
- main.py

##### Component Interactions and Data Flow
- Monitor → every X minutes → invokes match_list_scraper → fetches schedules → writes to MongoDB.
- Trigger_Jobs → polls upcoming/live matches → invokes match_detail_scraper → updates live data in MongoDB → emits “match_live” events to notify.py.
- WebSocket server listens for events from notify.py → pushes JSON payloads to connected clients.
- Clients subscribe over ws://localhost:6789 → receive real-time update messages.

##### Architectural Patterns and Design Principles
- Modular separation by concern: scraping, persistence, scheduling, and notification.
- Event-driven flow for real-time updates.
- Layered design: ingestion → persistence → notification.
- Use of job scheduler (APScheduler) ensures decoupled timed workflows.
- Asynchronous WebSocket server for low-latency push.

##### Detailed Elements to Capture
- Entry point: main.py spawns Scheduler and WebSocket server threads.
- monitor.py versus trigger_jobs.py roles.
- db_connection setup and query functions.
- clients subscribing channel and payload schema (match_id, event, score, timestamp).
- External service (Playwright) invocation.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Orchestration Entry
    MAIN["main.py"]:::orchestration
    REQ["requirements.txt"]:::external

    %% External Systems
    CW["Cricket Website\n(e.g. crex.live)"]:::external
    DBInstance["MongoDB Instance\n(localhost:27017)"]:::external
    Clients["WebSocket Clients\n(browsers, scripts)"]:::external

    %% Ingestion Layer
    subgraph "Ingestion Layer"
        MLS["match_list_scraper.py\n(Playwright)"]:::ingestion
        MDS["match_detail_scraper.py\n(Playwright)"]:::ingestion
    end

    %% Scheduler Layer
    subgraph "Scheduler Layer"
        MON["monitor.py\n(APScheduler)"]:::scheduling
        TRG["trigger_jobs.py\n(APScheduler)"]:::scheduling
    end

    %% Persistence Layer
    subgraph "Persistence Layer"
        CONNECTION["db_connection.py\n(motor/pymongo)"]:::persistence
        QUERIES["db_queries.py"]:::persistence
    end

    %% Notification Layer
    subgraph "Notification Layer"
        SERVER["server.py\n(websockets)"]:::websocket
        NOTIFY["notify.py"]:::websocket
    end

    %% Legend
    subgraph "Legend"
        L1["Ingestion Layer"]:::ingestion
        L2["Scheduler Layer"]:::scheduling
        L3["Persistence Layer"]:::persistence
        L4["Notification Layer"]:::websocket
        L5["External Systems / Clients"]:::external
    end

    %% Connections
    MAIN -->|spawns| MON
    MAIN -->|spawns| SERVER
    MAIN --> REQ

    MON -->|"Trigger match list scraper"| MLS
    MLS -->|"HTTP GET JSON"| CW
    MLS -->|"Insert/Update"| CONNECTION
    CONNECTION -->|"DB ops"| DBInstance

    TRG -->|"Trigger detail scraper"| MDS
    MDS -->|"HTTP GET JSON"| CW
    MDS -->|"Update"| QUERIES
    QUERIES -->|"DB ops"| DBInstance
    TRG -->|"Publish Event"| NOTIFY
    NOTIFY --> SERVER
    SERVER -->|"WebSocket Push"| Clients

    %% Click Events
    click MLS "https://github.com/dhruvrajsinh21/cricket_match_scraper/blob/main/scrapers/match_list_scraper.py"
    click MDS "https://github.com/dhruvrajsinh21/cricket_match_scraper/blob/main/scrapers/match_detail_scraper.py"
    click MON "https://github.com/dhruvrajsinh21/cricket_match_scraper/blob/main/scheduler/monitor.py"
    click TRG "https://github.com/dhruvrajsinh21/cricket_match_scraper/blob/main/scheduler/trigger_jobs.py"
    click CONNECTION "https://github.com/dhruvrajsinh21/cricket_match_scraper/blob/main/database/db_connection.py"
    click QUERIES "https://github.com/dhruvrajsinh21/cricket_match_scraper/blob/main/database/db_queries.py"
    click SERVER "https://github.com/dhruvrajsinh21/cricket_match_scraper/blob/main/websock/server.py"
    click NOTIFY "https://github.com/dhruvrajsinh21/cricket_match_scraper/blob/main/websock/notify.py"
    click MAIN "https://github.com/dhruvrajsinh21/cricket_match_scraper/blob/main/main.py"
    click REQ "https://github.com/dhruvrajsinh21/cricket_match_scraper/blob/main/requirements.txt"

    %% Styles
    classDef ingestion fill:#BBDEFB,stroke:#1E88E5,color:#0D47A1
    classDef scheduling fill:#FFE0B2,stroke:#FB8C00,color:#E65100
    classDef persistence fill:#C8E6C9,stroke:#43A047,color:#1B5E20
    classDef websocket fill:#E1BEE7,stroke:#8E24AA,color:#4A148C
    classDef external fill:#CFD8DC,stroke:#607D8B,color:#263238
    classDef orchestration fill:#B2EBF2,stroke:#00ACC1,color:#006064
```
