How cicada000/vv works
Main Components
• Data Processing & Preprocessing Pipeline:
• This module is responsible for processing the raw video assets and generating the searchable data.
• It includes the DataProcess folder scripts (compress_subtitle.py, create_db.py, import_subtitles.py, merge_screenshot.py, search_subtitles.py, and main.py) that generate JSON files with subtitle data.
• It also encompasses the subtitle extraction scripts (CutSubtitle.py and CutSubtitle_paddleocr.py) that use different OCR libraries (ddddocr and PaddleOCR).
• Face Recognition Engine:
• There are two implementations here: one using dlib in FaceRec.py and another using insightface in FaceRec_insightface.py.
• The modules generate_features.py and generate_features_insightface.py are used for precomputing face features, outputting files such as face_features.npz.
• This engine is integral during preprocessing to tag video frames with face features.
• Search Engine & Matching Module:
• Found in the “search” folder, this contains both “cloud” (with TypeScript, Node, and worker scripts) and “local” (Python-based) implementations.
• These modules are responsible for processing user queries and performing fuzzy text matching against the subtitle JSON files.
• API Backend:
• Located in the “api” folder, these scripts (bot.py, index.py, and the subfolder “search” that contains a Rust-based search engine) provide HTTP endpoints.
• The API layer exposes search functionality (e.g., handling query parameters such as query keywords, minimum text match ratio, and face similarity thresholds) and interacts with both the search engine and the data generated earlier.
• There is also an “api/subtitle_search_api” that likely wraps these functionalities into RESTful services.
• Web Frontend:
• The “Web” folder contains the client-side code (HTML, CSS, JS) that users interact with at https://vv.cicada000.work.
• It includes files such as index.html, db_search.js, and script.js along with configuration files (e.g., mapping.json) to display search results.
Relationships and Data Flow
• The data processing pipeline (including subtitle extraction and face recognition) is run either via main.py or as part of a preprocessing batch. Its output (e.g., JSON files in the “subtitle” directory) is stored in a database (or file-based storage) created via create_db.py.
• The face recognition component (using either dlib or insightface) annotates video frames to assist with verifying the “VV” faces during query matching.
• The search engine reads the subtitle data and leverages vector/text similarity algorithms (using both local Python scripts and Node/TS code in the cloud folder) for fuzzy matching.
• The API backend sits on top of the search engine and data store and exposes endpoints that the Web frontend calls. An example API call is provided in the README.
• The Web frontend (client user interface) sends search requests to the backend API, receives test results, and displays them to the user.
Architectural Patterns & Design Principles
• Layered Architecture: The project separates concerns among data ingestion/preprocessing, data retrieval/search, and client interfacing.
• Modular Design: Different recognition methods (dlib vs. insightface) and search approaches (local vs. cloud) are neatly separated into distinct modules.
• Extensibility: The design supports adding new video sources or recognition methods (as mentioned in the README) and can be extended or modified independently.
Technologies/Frameworks/Libraries
• Python is used for most processing, recognition (with ddddocr, PaddleOCR, dlib, insightface), and orchestration.
• Rust (in the api/search subdirectory) suggests use of performance-oriented search algorithms.
• Web technologies (HTML, CSS, JavaScript, TypeScript) power the frontend and parts of the “cloud” search.
• Deployment/configuration is facilitated by vercel.json and other configuration files.