How hoangduy181/website_monitoring works
Project Overview
• This project is a Python-based website monitoring system that checks websites for content changes, processes these changes, and sends Telegram notifications when differences are detected.
• Its primary goal is real-time monitoring with both single-run and scheduled checks, along with providing a baseline for comparison and handling content defacement detection.
Main Components
a. Configuration & Setup:
• Files involved: const.py, utils.py, README.md.
• Responsibilities: Store configuration constants like website URLs, Telegram bot token/chat id, and other global settings.
b. Baseline & Registration:
• File: register_url.py.
• Role: Sets up the initial state (baseline) for monitored websites.
c. Monitoring Core:
• Files: watcher.py and schedule_watch.py.
• Role: Core logic for detecting website changes; the watcher is run for one-time or periodic detection, whereas schedule_watch.py organizes recurring checks.
d. Notification & External Integration:
• Files: bot_helper.py, utils.py.
• Role: Manages interaction with the Telegram API using a bot for delivering notifications.
e. Content Storage & Data Artifacts:
• Directories: monitoring_htmls, monitoring_images.
• Additional Files: defacement_detector.pkl, ngram_model.py, hash_values.csv, capture_image.py.
• Role:
• These components handle snapshot storage (HTML captures, images) and may include intelligent processing for defacement detection using the ML model (defacement_detector.pkl) and n-gram analysis.
• Other helper scripts (e.g., redirect_and_not_found_check.py) support error handling and extra validation checks.
Relationships and Data Flow
• Bi-directional error handling feedback from network/website issues to the “Monitoring Engine”.
Architectural Patterns & Design Principles
• The separation of concerns is evident in dividing configuration, baseline setting, monitoring, notifications, and storage.
• The design follows a modular structure where each script or module handles a specific aspect of the system (e.g., baseline setup versus real-time monitoring).
• Data flow from initial configuration to the point where changes are detected, analyzed, stored, and then notified externally via Telegram.
Final Advice
• That even though the system is relatively small, you capture all essential modules and directional flows. This precision will help in understanding dependencies such as error handling around website connections and notifications.
• Document how scheduled tasks (via schedule_watch.py) interface with the monitoring engine and how the configuration parameters cascade into various functional flows.