How terrykwon/captivate works
Based on our analysis of the file tree and the README, this project is the server-side application for Captivate!, a system that ingests real-time video (and likely audio) streams to analyze parent–child interactions and provide language guidance. The system is designed to process multiple incoming video streams (from cameras) in parallel, perform various types of model inference (e.g., gaze following, face/object detection), and then combine these insights into a joint attention distribution for contextual language guidance. The architecture is modular—dividing responsibilities among model processing, guidance generation (using CSV-based sentence guidance), and server communication (using WebSockets)—and relies on Docker for containerized deployment (as seen via Dockerfiles and docker-compose).
Main Components
• Modelserver – This is the core processing component:
• A main process (run_server.py) that orchestrates parallel processing.
• Worker processes, each handling one video stream’s pipeline (as noted for three streams in the paper, but configurable).
• Subcomponents include:
▫ Base Modules: Face detection, face recognition, gaze following, object detection/segmentor (found in modelserver/base).
▫ Model pipelines for visual attention (located in modelserver/models/visual-attention/visatt) with support libraries (e.g., pytorch_convolutional_rnn).
▫ Unimodal processing modules (e.g., visual, imagestream, audial) that perform modality-specific tasks.
▫ Guidance Module: Processes and loads guidance sentences from CSV files (in modelserver/guidance) that are used for generating context-specific language recommendations.
▫ Utility and visualization components that support debugging and output formatting.
• Data Integration & Aggregation – Illustrate the shared queue in the main process where outputs from individual model workers are gathered and then combined into a joint attention distribution.
• Server/Communication Layer – Mark out the real-time communication using WebSockets (as mentioned in the README) that relay the combined outputs to the client.
• Videoserver – Represent the supplementary component that likely manages or proxies video streaming to clients:
• Docker configuration and Nginx setup (nginx.conf) indicating reverse proxy or load balancing functionality.
• External Dependencies – Note that the system integrates external models (referenced repositories for gaze following, face detection, object detection, and language processing).
Relationships and Interactions
• That each camera’s stream goes to a separate worker process which performs the model inference.
• That the main process reads from this queue, integrates/combines the outputs, and computes the joint attention distribution.
• Final inference output to the WebSocket server component that sends data to the client application.
• Optionally, indicate where Docker container boundaries exist (i.e., Modelserver docked container, Videoserver container).
Architectural Patterns and Design Principles
• The system follows a modular and parallel processing design, where identical processing pipelines (for each camera stream) enable scalability and flexibility.
• Separation of concerns:
• Model Inference vs. Data Aggregation vs. Communication.
• The use of multiple Dockerfiles and docker-compose suggests containerized microservice-like components.
• Integration of external libraries/repositories as plug-ins to extend core functionality.
Technologies and Frameworks
• Docker (indicated by Dockerfiles and docker-compose.yml) for containerization.
• Python as the main programming language (given the .py files, and use of environment and setup scripts).
• PyTorch and specialized libraries for convolutions and RNNs (inside the visual attention module).
• Nginx in the Videoserver for video stream management or reverse proxying.
• WebSockets for real-time data communications between the server and the client.