How mozhang-ah/aiwatchdog works
Main Components
• Video Input Source: This is the entry point for either local video files or RTSP streams.
• Video Server (video_server.py): The main executable that handles video stream ingestion. It passes incoming video frames to the processing pipeline.
• Video Processor & Frame Buffer: Represent the video processing unit that splits the video into frames and stores them temporarily (e.g., the “帧缓冲区” as shown in the README flowchart).
• Multi-modal Analyzer (multi_modal_analyzer.py): This component takes batches of video frames from the buffer and performs analysis using multi-modal techniques, integrating models (which might be configured via config.py) to generate content descriptions.
• Content Description Generator: A module (implied in the README flow) that creates text descriptions of what’s happening in the scene.
• Abnormal Behavior Detector: Processes the generated content to determine if the behavior is abnormal.
• Pre-warning Service: This service, upon detecting anomalies, pushes alerts via WebSocket (using URLs provided in the README) and triggers actions like saving abnormal video fragments and screenshots.
• Historical Record Storage: For non-alarming cases, event records are stored (this might be visible through the “video_histroy_info.txt” file).
• (Optional) RAG Pipeline: If enabled, parts of the system interact with a vector database (e.g., Milvus) to store and later retrieve text vectors. This component handles the “文本向量化” of content descriptions, supports smart Q&A, and connects user queries with relevant information.
Relationships and Data Flow
• The Analyzer outputs processed data to the Content Description Generator.
• Next, indicate a split: one path sends generated text into the Abnormal Behavior Detector.
– If an anomaly is detected, it flows into the Pre-warning Service, which:
▪ Sends WebSocket messages to clients.
▪ Triggers abnormal video and screenshot storage.
– If no anomaly is detected, then the video data moves to Historical Record Storage for periodic summarization.
▪ Content descriptions from the generator (and data from the pre-warning and historical summarization components) are vectorized and stored in a Vector Database.
▪ User queries are processed through a text vectorization step and then the vector database is used to fetch similar cases leading to a smart question-answer module.
Architectural Patterns and Design Principles
• Separation of concerns: The system is modular, with clear division between video capture, processing, analysis, and alerting.
• Real-time processing is central; hence using streaming and buffering concepts.
• Optional components (like the RAG system) should be noted as pluggable modules.
• The system employs event-driven communication (especially for WebSocket alerts).
Technologies and Frameworks
• Python is the primary language (as indicated by .py files and requirements.txt).
• WebSocket technology is used for alert distribution.
• For multi-modal analysis, the project likely relies on deep learning models (configured via config.py).
• If RAG is enabled, a vector database (e.g., Milvus) is expected.
• Utility functions and prompt configurations (in prompt.py and utility.py) help in processing the video frames and generating text descriptions.
Additional Elements
• Where configuration (config.py) and customization of prompts (prompt.py) come into play.
• External interactions: WebSocket clients (for alert push) and potentially external vector database services.
• If possible, add deployment details like how the Python environment (requirements.txt) and logs (code.log) fit within the operational ecosystem.
This approach ensures that all essential modules and interactions are captured, making it easier for stakeholders to grasp and for teams to implement and maintain the system.