# inmotion-health/ctrlability architecture

> Architecture diagram of the GitHub repository inmotion-health/ctrlability, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/inmotion-health/ctrlability
- Repository: https://github.com/inmotion-health/ctrlability

## Overview

##### Main System Type and Purpose
- CTRLABILITY is a desktop application (CLI + optional GUI) that captures webcam video and microphone audio, processes those streams to detect gestures/voice, and translates them into OS input commands (mouse, keyboard).
- It comprises two Python packages:
– ctrlability: the core pipeline (streams → processors → triggers → actions)
– ctrlability_ui: an MVC GUI on top of the core that allows configuration, live preview, and logging.

##### Top-Level Components
A. External Interfaces
1. Webcam (video capture)
2. Microphone (audio capture)
3. OS Input API (to inject mouse/keyboard events)
4. External libs: FFmpeg (video/audio I/O), MediaPipe (landmark detection)
B. Configuration
– config.yaml / examples/*.yaml
– ctrlability/core/config_parser.py (parses into in-memory config)
C. Core Pipeline (ctrlability package)
1. Bootstrapper (ctrlability/core/bootstrapper.py) – reads config, instantiates streams, processors, triggers, actions
2. Streams module (ctrlability/streams/video.py) – continuously produces raw frames/audio
3. StreamHandler (ctrlability/core/stream_handler.py) – drives the pipeline loop
4. Processors (ctrlability/processors/) – e.g. face_landmarks, hand_landmarks, audio processing
5. MappingEngine (ctrlability/core/mapping_engine.py) – orchestrates triggering logic
6. Triggers (ctrlability/triggers/) – evaluate conditions (e.g. landmark distance, facial expression)
7. Actions (ctrlability/actions/) – perform OS commands: key_command, mouse_move, mouse_click, logging
D. GUI Layer (ctrlability_ui package)
1. EntryPoint (ctrlability_ui/__main__.py) – launches GUI
2. MVC Pattern
- Models (ctrlability_ui/models/) – hold application state (current config, live data)
- Views (ctrlability_ui/views/) – UI components: video preview, logs, settings panels
- Controllers (ctrlability_ui/controllers/) – handle user events (load config, start/stop stream), update models
3. State Observer (ctrlability_ui/patterns/state_observer.py) – propagates pipeline events to views
4. ProcessThread (ctrlability_ui/threads/process_thread.py) – runs core pipeline in background

##### Relationships and Data Flow
- Data flows from External Interfaces → Streams → StreamHandler → Processors → MappingEngine → Triggers → Actions → OS.
- ConfigParser feeds Bootstrapper with component parameters.
- In GUI mode, ProcessThread encapsulates the core pipeline; outputs and logs are emitted via StateObserver into Models, and Controllers/Views subscribe to model changes for real-time updates.
- Controllers in UI can reconfigure or restart the core pipeline by invoking Bootstrapper with new config.

##### Architectural Patterns & Design Principles
- Pipeline Pattern: sequential processing stages (streams → processors → triggers → actions)
- MVC for GUI with Observer for live updates
- Modular separation: each processor/trigger/action in its own module for extensibility
- Configuration-driven instantiation (no hardcoded pipelines)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Layer
    subgraph "External Layer"
        direction TB
        Webcam["Webcam"]:::external
        Mic["Microphone"]:::external
        FFmpeg["FFmpeg"]:::external
        MediaPipe["MediaPipe"]:::external
        OSAPI["OS Input API"]:::external
    end

    %% Configuration Layer
    subgraph "Configuration" 
        direction TB
        ConfigFile["config.yaml / examples/*.yaml"]:::config
        ConfigParser["Config Parser"]:::config
    end

    %% Core Pipeline Layer
    subgraph "Core Pipeline" 
        direction TB
        BS["Bootstrapper"]:::core
        Streams["Streams\n(video/audio)"]:::core
        StreamHandler["Stream Handler"]:::core
        subgraph "Processors"
            direction TB
            FL["face_landmarks"]:::core
            HL["hand_landmarks"]:::core
            HOL["holistic_landmarks"]:::core
            PL["pose_landmarks"]:::core
            SD["signal_divider"]:::core
            VFG["video_frame_grabber"]:::core
            LE["landmark_eurofilter"]:::core
            LNV["landmark_normal_vector"]:::core
            FEC["facial_expression_classifier"]:::core
        end
        ME["Mapping Engine"]:::core
        subgraph "Triggers"
            direction TB
            ACT1["absolute_cursor_control"]:::core
            ACT2["relative_cursor_control"]:::core
            ACT3["facial_expression_trigger"]:::core
            ACT4["landmark_distance"]:::core
            ACT5["region_of_interest"]:::core
            ACT6["throughput"]:::core
        end
        subgraph "Actions"
            direction TB
            A1["key_command"]:::core
            A2["mouse_move"]:::core
            A3["mouse_click"]:::core
            A4["logging_action"]:::core
        end
    end

    %% GUI Layer
    subgraph "GUI Layer (Optional)"
        direction TB
        EntryPoint["GUI EntryPoint"]:::gui
        ProcessThread["Process Thread"]:::gui
        StateObserver["State Observer"]:::gui
        Controllers["Controllers"]:::gui
        Models["Models"]:::gui
        Views["Views"]:::gui
    end

    %% Flows: external to core
    Webcam -->|video frames| Streams
    Mic -->|audio samples| Streams
    FFmpeg --> Streams
    MediaPipe --> Streams

    %% Config flow
    ConfigFile --> ConfigParser
    ConfigParser --> BS
    click ConfigParser "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/core/config_parser.py"

    %% Core pipeline sequence
    BS --> Streams
    click BS "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/core/bootstrapper.py"
    Streams --> StreamHandler
    click Streams "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/streams/video.py"
    click StreamHandler "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/core/stream_handler.py"
    StreamHandler --> FL
    StreamHandler --> HL
    StreamHandler --> HOL
    StreamHandler --> PL
    StreamHandler --> SD
    StreamHandler --> VFG
    StreamHandler --> LE
    StreamHandler --> LNV
    StreamHandler --> FEC
    click FL "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/processors/face_landmarks.py"
    click HL "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/processors/hand_landmarks.py"
    click HOL "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/processors/holistic_landmarks.py"
    click PL "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/processors/pose_landmarks.py"
    click SD "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/processors/signal_divider.py"
    click VFG "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/processors/video_frame_grabber.py"
    click LE "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/processors/landmark_eurofilter.py"
    click LNV "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/processors/landmark_normal_vector.py"
    click FEC "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/processors/facial_expression_classifier.py"

    FL --> ME
    HL --> ME
    HOL --> ME
    PL --> ME
    SD --> ME
    VFG --> ME
    LE --> ME
    LNV --> ME
    FEC --> ME
    click ME "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/core/mapping_engine.py"

    ME --> ACT1
    ME --> ACT2
    ME --> ACT3
    ME --> ACT4
    ME --> ACT5
    ME --> ACT6
    click ACT1 "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/triggers/absolute_cursor_control.py"
    click ACT2 "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/triggers/relative_cursor_control.py"
    click ACT3 "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/triggers/facial_expression_trigger.py"
    click ACT4 "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/triggers/landmark_distance.py"
    click ACT5 "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/triggers/region_of_interest.py"
    click ACT6 "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/triggers/throughput.py"

    ACT1 --> A1
    ACT2 --> A2
    ACT3 --> A1
    ACT4 --> A3
    ACT5 --> A2
    ACT6 --> A4
    click A1 "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/actions/key_command.py"
    click A2 "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/actions/mouse_move.py"
    click A3 "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/actions/mouse_click.py"
    click A4 "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/actions/logger.py"

    A1 --> OSAPI
    A2 --> OSAPI
    A3 --> OSAPI
    A4 --> OSAPI

    click OSAPI "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability/helpers/mousectrl.py"

    %% GUI interactions
    EntryPoint --> ProcessThread
    click EntryPoint "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability_ui/__main__.py"
    ProcessThread --> BS
    click ProcessThread "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability_ui/threads/process_thread.py"
    ProcessThread --> StateObserver
    click StateObserver "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability_ui/patterns/state_observer.py"
    StateObserver --> Models
    click Models "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability_ui/models/ctrlability_model.py"
    Models --> Views
    click Views "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability_ui/views/main_view.py"
    Controllers --> BS
    click Controllers "https://github.com/inmotion-health/ctrlability/blob/main/ctrlability_ui/controllers/ctrlability_controller.py"
    Controllers --> Models
    Views --> Controllers
    Models --> Views

    %% Configuration examples
    ConfigFile -->|example configs| ExampleConfigs["examples/*.yaml"]:::config
    click ExampleConfigs "https://github.com/inmotion-health/ctrlability/blob/main/examples/config-expressions.yaml"

    %% Styles
    classDef external fill:#cce5ff,stroke:#0066cc,color:#003366;
    classDef config fill:#fff2cc,stroke:#cc9900,color:#665200;
    classDef core fill:#d5f5e3,stroke:#27ae60,color:#145a32;
    classDef gui fill:#fdebd0,stroke:#e67e22,color:#844f00;
```
