# hoangduy181/website_monitoring architecture

> Architecture diagram of the GitHub repository hoangduy181/website_monitoring, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hoangduy181/website_monitoring
- Repository: https://github.com/hoangduy181/website_monitoring

## Overview

##### Project Overview
- This project is a Python-based website monitoring system that checks websites for content changes, processes these changes, and sends Telegram notifications when differences are detected.
- Its primary goal is real-time monitoring with both single-run and scheduled checks, along with providing a baseline for comparison and handling content defacement detection.

##### Main Components
a. Configuration & Setup:
- Files involved: const.py, utils.py, README.md.
- Responsibilities: Store configuration constants like website URLs, Telegram bot token/chat id, and other global settings.

b. Baseline & Registration:
- File: register_url.py.
- Role: Sets up the initial state (baseline) for monitored websites.

c. Monitoring Core:
- Files: watcher.py and schedule_watch.py.
- Role: Core logic for detecting website changes; the watcher is run for one-time or periodic detection, whereas schedule_watch.py organizes recurring checks.

d. Notification & External Integration:
- Files: bot_helper.py, utils.py.
- Role: Manages interaction with the Telegram API using a bot for delivering notifications.

e. Content Storage & Data Artifacts:
- Directories: monitoring_htmls, monitoring_images.
- Additional Files: defacement_detector.pkl, ngram_model.py, hash_values.csv, capture_image.py.
- Role:
- These components handle snapshot storage (HTML captures, images) and may include intelligent processing for defacement detection using the ML model (defacement_detector.pkl) and n-gram analysis.
- Other helper scripts (e.g., redirect_and_not_found_check.py) support error handling and extra validation checks.

##### Relationships and Data Flow
- Bi-directional error handling feedback from network/website issues to the “Monitoring Engine”.

##### Architectural Patterns & Design Principles
- The separation of concerns is evident in dividing configuration, baseline setting, monitoring, notifications, and storage.
- The design follows a modular structure where each script or module handles a specific aspect of the system (e.g., baseline setup versus real-time monitoring).
- Data flow from initial configuration to the point where changes are detected, analyzed, stored, and then notified externally via Telegram.

##### Final Advice
- That even though the system is relatively small, you capture all essential modules and directional flows. This precision will help in understanding dependencies such as error handling around website connections and notifications.
- Document how scheduled tasks (via schedule_watch.py) interface with the monitoring engine and how the configuration parameters cascade into various functional flows.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Configuration & Setup"
         A1["const.py"]:::config
         A2["utils.py"]:::config
         A3["README.md"]:::config
         A0["Configuration Settings"]:::config
         A1 --> A0
         A2 --> A0
         A3 --> A0
    end

    subgraph "Baseline Registration"
         B1["register_url.py"]:::baseline
    end

    subgraph "Monitoring Core"
         M1["watcher.py"]:::monitoring
         M2["schedule_watch.py"]:::monitoring
         M0["Monitoring Engine"]:::monitoring
         M1 --> M0
         M2 --> M0
         N1["Network/Website Issues"]:::monitoring
         M0 <--> N1
    end

    subgraph "Notification & External Integration"
         N2["bot_helper.py"]:::notification
         NS["Telegram Notification Service"]:::notification
         N2 --> NS
    end

    subgraph "Content Storage"
         CS1["monitoring_htmls"]:::storage
         CS2["monitoring_images"]:::storage
         CS0["Content Storage"]:::storage
         CS1 --> CS0
         CS2 --> CS0
    end

    subgraph "Analysis/Detection Module"
         A_D1["defacement_detector.pkl"]:::analysis
         A_D2["ngram_model.py"]:::analysis
         A_D3["hash_values.csv"]:::analysis
         A_D4["capture_image.py"]:::analysis
         A_D5["redirect_and_not_found_check.py"]:::analysis
         AD0["Analysis/Detection Module"]:::analysis
         A_D1 --> AD0
         A_D2 --> AD0
         A_D3 --> AD0
         A_D4 --> AD0
         A_D5 --> AD0
    end

    %% Top-level Connections
    A0 -->|"provides_config"| B1
    A0 -->|"provides_config"| M0
    B1 -->|"stores_baseline"| CS0
    M0 -->|"saves_snapshot"| CS0
    M0 -->|"sends_for_analysis"| AD0
    M0 -->|"sends_notification"| NS

    %% Click Events
    click A1 "https://github.com/hoangduy181/website_monitoring/blob/main/const.py"
    click A2 "https://github.com/hoangduy181/website_monitoring/blob/main/utils.py"
    click A3 "https://github.com/hoangduy181/website_monitoring/blob/main/README.md"
    click B1 "https://github.com/hoangduy181/website_monitoring/blob/main/register_url.py"
    click M1 "https://github.com/hoangduy181/website_monitoring/blob/main/watcher.py"
    click M2 "https://github.com/hoangduy181/website_monitoring/blob/main/schedule_watch.py"
    click N2 "https://github.com/hoangduy181/website_monitoring/blob/main/bot_helper.py"
    click CS1 "https://github.com/hoangduy181/website_monitoring/tree/main/monitoring_htmls"
    click CS2 "https://github.com/hoangduy181/website_monitoring/tree/main/monitoring_images"
    click A_D1 "https://github.com/hoangduy181/website_monitoring/blob/main/defacement_detector.pkl"
    click A_D2 "https://github.com/hoangduy181/website_monitoring/blob/main/ngram_model.py"
    click A_D3 "https://github.com/hoangduy181/website_monitoring/blob/main/hash_values.csv"
    click A_D4 "https://github.com/hoangduy181/website_monitoring/blob/main/capture_image.py"
    click A_D5 "https://github.com/hoangduy181/website_monitoring/blob/main/redirect_and_not_found_check.py"

    %% Styles
    classDef config fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef baseline fill:#F4A460,stroke:#000,stroke-width:2px;
    classDef monitoring fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef notification fill:#FFCCCB,stroke:#000,stroke-width:2px;
    classDef storage fill:#FFA500,stroke:#000,stroke-width:2px;
    classDef analysis fill:#D8BFD8,stroke:#000,stroke-width:2px;
```
