How inmotion-health/ctrlability works
Main System Type and Purpose
• CTRLABILITY is a desktop application (CLI + optional GUI) that captures webcam video and microphone audio, processes those streams to detect gestures/voice, and translates them into OS input commands (mouse, keyboard).
• It comprises two Python packages:
– ctrlability: the core pipeline (streams → processors → triggers → actions)
– ctrlability_ui: an MVC GUI on top of the core that allows configuration, live preview, and logging.
Top-Level Components
A. External Interfaces
1. Webcam (video capture)
2. Microphone (audio capture)
3. OS Input API (to inject mouse/keyboard events)
4. External libs: FFmpeg (video/audio I/O), MediaPipe (landmark detection)
B. Configuration
– config.yaml / examples/*.yaml
– ctrlability/core/config_parser.py (parses into in-memory config)
C. Core Pipeline (ctrlability package)
1. Bootstrapper (ctrlability/core/bootstrapper.py) – reads config, instantiates streams, processors, triggers, actions
2. Streams module (ctrlability/streams/video.py) – continuously produces raw frames/audio
3. StreamHandler (ctrlability/core/stream_handler.py) – drives the pipeline loop
4. Processors (ctrlability/processors/) – e.g. face_landmarks, hand_landmarks, audio processing
5. MappingEngine (ctrlability/core/mapping_engine.py) – orchestrates triggering logic
6. Triggers (ctrlability/triggers/) – evaluate conditions (e.g. landmark distance, facial expression)
7. Actions (ctrlability/actions/) – perform OS commands: key_command, mouse_move, mouse_click, logging
D. GUI Layer (ctrlability_ui package)
1. EntryPoint (ctrlability_ui/__main__.py) – launches GUI
2. MVC Pattern
• Models (ctrlability_ui/models/) – hold application state (current config, live data)
• Views (ctrlability_ui/views/) – UI components: video preview, logs, settings panels
• Controllers (ctrlability_ui/controllers/) – handle user events (load config, start/stop stream), update models
3. State Observer (ctrlability_ui/patterns/state_observer.py) – propagates pipeline events to views
4. ProcessThread (ctrlability_ui/threads/process_thread.py) – runs core pipeline in background
Relationships and Data Flow
• Data flows from External Interfaces → Streams → StreamHandler → Processors → MappingEngine → Triggers → Actions → OS.
• ConfigParser feeds Bootstrapper with component parameters.
• In GUI mode, ProcessThread encapsulates the core pipeline; outputs and logs are emitted via StateObserver into Models, and Controllers/Views subscribe to model changes for real-time updates.
• Controllers in UI can reconfigure or restart the core pipeline by invoking Bootstrapper with new config.
Architectural Patterns & Design Principles
• Pipeline Pattern: sequential processing stages (streams → processors → triggers → actions)
• MVC for GUI with Observer for live updates
• Modular separation: each processor/trigger/action in its own module for extensibility
• Configuration-driven instantiation (no hardcoded pipelines)