# mohammad-muzahid/ironman_interface_detection architecture

> Architecture diagram of the GitHub repository mohammad-muzahid/ironman_interface_detection, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mohammad-muzahid/ironman_interface_detection
- Repository: https://github.com/mohammad-muzahid/ironman_interface_detection

## Overview

This is a **single-process Python application** that builds a **futuristic “Ironman-style” HUD** for **real-time visual understanding** using a **VLM (Vision-Language Model) technique**. The screenshots imply:
- Live camera input (or images/video)
- Object detection / region identification
- VLM-based reasoning/labeling and/or description generation
- UI overlay / visualization
- Optional speech input/output loop (voice commands and/or spoken descriptions)

So the “system” is less about distributed services and more about a **pipeline architecture** inside one runtime, with heavy dependencies on **ML models** and **I/O devices** (camera, mic, display).

##### What the file tree tells you (map files → architectural components)
Top-level files strongly suggest “modules as components,” with `main.py` orchestrating:

- **Entry / Orchestrator**
- `main.py` — likely the run loop: capture frame → detect → VLM → generate text → render overlay → (optional) speech.
- `config.py` — central configuration (model names, thresholds, camera index, etc.).

- **Vision / Detection**
- `object_detector.py` — classic CV/ML detector wrapper (could be YOLO/Detectron/etc.).
- `VLMProcessor.py` — core VLM inference wrapper (image + prompt → text/logits).
- `advanced_vlm.py` — enhanced VLM logic (prompt chaining, multi-step reasoning, caching, better parsing).

- **Language / Content generation**
- `description_generator.py` — converts detections + VLM outputs into user-facing text (“Ironman HUD labels,” summaries).

- **Speech / Audio**
- `speech_processor.py` — microphone input, speech-to-text, text-to-speech, command parsing.

- **Presentation**

- **Packaging / Environment**
- `requirements.txt` — external libs (ML, CV, audio, etc.).
- `setup.py` — packaging/install.
- `README.md` — high-level intent and visuals.

This structure suggests a **modular pipeline** with a clear separation of concerns (capture/vision/VLM/text/speech/render).

For accuracy and usefulness, produce **three views** (principal-engineer-friendly):

External actors/resources:
- **User**
- **Camera / Video Source**
- **Microphone / Speaker** (if speech is used)
- **Display / Windowing system**
- **Model source** (local checkpoints or HuggingFace/downloaded weights)
- **GPU/CPU runtime** (CUDA/Metal/etc., if relevant)

This answers: “Where does data come from/go to?”

Even though it’s one Python process, treat major modules as “containers”:
- /Input subsystem
- Object Detection subsystem
- VLM subsystem (basic + advanced)
- Description/Text subsystem
- Speech subsystem
- Visualization/HUD subsystem
- Config/Utilities

This answers: “What are the big blocks and how do they connect?”

This is the most important here. Show:
- Frame acquisition loop
- ROI crops / prompts into VLM
- VLM outputs: labels / attributes / reasoning text
- Description generator produces HUD strings
- Visualizer renders overlay and outputs frames
- Speech processor inserts commands into the loop (e.g., change mode, query object) and/or speaks results

This answers: “What is computed at each stage and what data structure moves along the pipeline?”

##### `main.py` (Runtime Controller)
- Initializes modules using `config.py`
- Owns the main event loop and timing (FPS, async calls, batching)
- Routes outputs between modules

##### `config.py` (Configuration/Policy)
- Model selection, thresholds, prompt templates, device settings, UI settings

##### `object_detector.py` (Detection Engine)
- Input: frame
- If it uses a deep model, represent “Detector Weights/Model” as an external dependency bubble.

##### `VLMProcessor.py` (VLM Inference Core)
- Input: frame/ROI + prompt
- Output: text/labels/attributes
- Dependency: “VLM weights / tokenizer / preprocessor”

##### `advanced_vlm.py` (VLM Orchestration Layer)
- Prompt chaining, structured parsing, reasoning steps, multiple calls, caching

##### `description_generator.py` (Narrative/HUD Text Builder)
- Input: detections + VLM outputs + mode
- Output: final display strings / structured HUD elements

##### `speech_processor.py` (Speech I/O + Commands)
- Input: microphone audio
- Output: commands → `main.py` (control signals)
- Optional: text-to-speech from descriptions

##### `visualizer.py` (HUD Renderer)
- Input: frame + detections + HUD text
- Output: rendered frame/window stream

##### object_detector.py → main.py (detections)
4. **main.py → advanced_vlm.py / VLMProcessor.py** (ROI crops + prompts; optionally full frame)
##### visualizer.py → Display Window / Output Stream

- **speech_processor.py → main.py** (commands: “scan”, “focus on X”, “increase threshold”, “toggle mode”)
- **main.py → speech_processor.py** (optional TTS: speak summary)
- **config.py → all modules** (initialization + constants)
- **advanced_vlm.py → VLMProcessor.py** (calls)

- “Model Weights / Checkpoints” connected to detector + VLM
- “Hardware Acceleration (GPU)” connected to detector + VLM

- **Pipeline / staged processing** (frame → detect → reason → describe → render)
- **Orchestrator / Facade** (`main.py` coordinating)
- **Strategy / pluggable model backends** (basic VLM vs `advanced_vlm`)
- **Event-driven controls** (speech commands affecting runtime state)

- **Blue rectangles**: internal Python modules (`main.py`, `object_detector.py`, etc.)
- **Orange parallelograms**: I/O devices (Camera, Mic, Speaker, Display)
- Thick = main data path (frames, detection results)
- Thin = control signals (commands, config updates)

- `frame (H×W×3)`
- `prompt + ROI`
- `vlm_text / parsed_attributes`
- `HUD_overlay عناصر` (text, reticles, lines)

##### What “most accurate” means here (how to validate against the repo)
Because there are no subdirectories and the README is visual, accuracy comes from:
1. **Open each module and identify the public entrypoints** (e.g., `detect()`, `process_frame()`, `generate_description()`, `render()`).
##### Confirm runtime mode
- Single-threaded loop vs async vs multiprocessing (if present)
- Streaming vs single-image
##### Confirm what `advanced_vlm.py` actually adds
- multiple VLM calls per frame?
- caching?
- structured output parsing?

Those realities with:
- A loop marker around the pipeline (per-frame loop)
- Optional branches (e.g., “speech enabled?” “advanced mode?”)

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =========================
  %% Level 0: Context (Actors / External I/O)
  %% =========================
  subgraph "Level 0 — Context (External Actors & Resources)"
    direction TB

    User["User"]:::external

    Camera[/"Camera / Video Source"/]:::io
    Mic[/"Microphone"/]:::io
    Speaker[/"Speaker"/]:::io
    Display[/"Display / Window System"/]:::io

    subgraph "External Model Assets"
      direction TB
      DetectorWeights[("Detector Weights / Checkpoints")]:::model
      VLMWeights[("VLM Weights / Tokenizer / Preprocessor")]:::model
      VLMCACHE[("VLM Cache (optional)")]:::model
    end

    subgraph "Runtime & Libraries (External)"
      direction TB
      HW["Hardware Acceleration (GPU/CPU)"]:::runtime
      Libs["Libraries: OpenCV, Torch, Transformers, Audio I/O"]:::runtime
    end
  end

  %% =========================
  %% Level 1/2: Single-process container + pipeline
  %% =========================
  subgraph "Level 1/2 — Single Python Process (Pipeline + Orchestration)"
    direction TB

    subgraph "Swimlane 1 — Inputs / Outputs"
      direction TB
      InCam[/"Frames In"/]:::io
      InMic[/"Audio In"/]:::io
      OutSpk[/"Audio Out"/]:::io
      OutDisp[/"Rendered Frames Out"/]:::io
    end

    subgraph "Swimlane 2 — Core Runtime"
      direction TB
      CFG["config.py\nConfiguration / Policy"]:::module
      MAIN["main.py\nRuntime Controller / Orchestrator\n(FPS loop + mode/state)"]:::orchestrator
      MODE["Mode / State Machine\n(scan / focus / threshold / overlays)"]:::internal
    end

    subgraph "Swimlane 3 — Processing Modules (Staged Pipeline)"
      direction TB

      DET["object_detector.py\nDetection Engine"]:::module

      subgraph "VLM Subsystem (Strategy: basic vs advanced)"
        direction TB
        ADV["advanced_vlm.py\nVLM Orchestration\n(prompt chaining + parsing + caching)"]:::module
        VLM["VLMProcessor.py\nVLM Inference Core"]:::module
      end

      DESC["description_generator.py\nNarrative / HUD Text Builder"]:::module
      VIS["visualizer.py\nHUD Renderer / Visualization"]:::module

      SP["speech_processor.py\nSpeech I/O + Commands\n(STT/TTS)"]:::module
    end

    %% Data artifacts (for labeling payloads)
    Frame["Data: frame (H×W×3)"]:::artifact
    Dets["Data: detections\n[box, cls, conf]"]:::artifact
    PromptROI["Data: prompt + ROI crops"]:::artifact
    VLMOut["Data: vlm_text / parsed_attributes"]:::artifact
    HUD["Data: HUD_overlay elements\n(text, reticles, lines)"]:::artifact
    Cmds["Data: commands\n(scan/focus/toggle/threshold)"]:::artifact
  end

  %% =========================
  %% Connections — Context to app I/O
  %% =========================
  User -->|"uses"| Camera
  User -->|"speaks_to"| Mic
  Display -->|"shows_to"| User
  Speaker -->|"plays_to"| User

  Camera -->|"frames"| InCam
  Mic -->|"audio_stream"| InMic
  OutDisp -->|"video_stream"| Display
  OutSpk -->|"tts_audio"| Speaker

  %% =========================
  %% Connections — Main pipeline (thick)
  %% =========================
  InCam ==>|"frame (H×W×3)"| MAIN
  MAIN ==>|"frame"| DET
  DET ==>|"detections: [box, cls, conf]"| MAIN

  MAIN ==>|"ROI + prompt"| ADV
  ADV ==>|"calls"| VLM
  VLM ==>|"vlm_text"| ADV
  ADV ==>|"vlm_text / parsed_attributes"| MAIN

  MAIN ==>|"detections + vlm_result + mode"| DESC
  DESC ==>|"HUD_overlay elements"| MAIN

  MAIN ==>|"frame + detections + HUD_overlay"| VIS
  VIS ==>|"rendered_frame"| OutDisp

  %% =========================
  %% Control / feedback (thin)
  %% =========================
  InMic -->|"audio"| SP
  SP -->|"commands"| MAIN
  MAIN -->|"state_updates"| MODE
  MODE -->|"mode"| MAIN

  MAIN -->|"optional_tts(text)"| SP
  SP -->|"audio_out"| OutSpk

  %% Config fan-out (thin)
  CFG -.->|"init/constants"| MAIN
  CFG -.->|"thresholds/models"| DET
  CFG -.->|"prompt_templates"| ADV
  CFG -.->|"device/settings"| VLM
  CFG -.->|"ui/settings"| VIS
  CFG -.->|"voice/settings"| SP
  CFG -.->|"formatting/modes"| DESC

  %% =========================
  %% External dependencies (dashed)
  %% =========================
  DetectorWeights -.->|"loads"| DET
  VLMWeights -.->|"loads"| VLM
  VLMCACHE -.->|"optional"| ADV

  HW -.->|"accelerates"| DET
  HW -.->|"accelerates"| VLM
  Libs -.->|"provides"| DET
  Libs -.->|"provides"| VLM
  Libs -.->|"provides"| VIS
  Libs -.->|"provides"| SP

  %% =========================
  %% Data artifacts mapping (dotted to stages)
  %% =========================
  MAIN -.->|"produces/consumes"| Frame
  DET -.->|"produces"| Dets
  ADV -.->|"consumes"| PromptROI
  ADV -.->|"produces"| VLMOut
  DESC -.->|"produces"| HUD
  SP -.->|"produces"| Cmds

  %% =========================
  %% Pattern annotations
  %% =========================
  PIPE["Pattern: Pipeline / staged processing"]:::note
  ORCH["Pattern: Orchestrator / Facade (main.py)"]:::note
  STRAT["Pattern: Strategy (basic VLM vs advanced orchestration)"]:::note
  EVT["Pattern: Event-driven controls (speech commands)"]:::note

  PIPE -.-> MAIN
  ORCH -.-> MAIN
  STRAT -.-> ADV
  STRAT -.-> VLM
  EVT -.-> SP

  %% =========================
  %% Click Events (from component_mapping)
  %% =========================
  click MAIN "https://github.com/mohammad-muzahid/ironman_interface_detection/blob/main/main.py"
  click CFG "https://github.com/mohammad-muzahid/ironman_interface_detection/blob/main/config.py"
  click DET "https://github.com/mohammad-muzahid/ironman_interface_detection/blob/main/object_detector.py"
  click VLM "https://github.com/mohammad-muzahid/ironman_interface_detection/blob/main/VLMProcessor.py"
  click ADV "https://github.com/mohammad-muzahid/ironman_interface_detection/blob/main/advanced_vlm.py"
  click DESC "https://github.com/mohammad-muzahid/ironman_interface_detection/blob/main/description_generator.py"
  click SP "https://github.com/mohammad-muzahid/ironman_interface_detection/blob/main/speech_processor.py"
  click VIS "https://github.com/mohammad-muzahid/ironman_interface_detection/blob/main/visualizer.py"
  click Libs "https://github.com/mohammad-muzahid/ironman_interface_detection/blob/main/requirements.txt"
  click HW "https://github.com/mohammad-muzahid/ironman_interface_detection/blob/main/setup.py"
  click PIPE "https://github.com/mohammad-muzahid/ironman_interface_detection/blob/main/README.md"

  %% =========================
  %% Styles (color is critical)
  %% =========================
  classDef orchestrator fill:#1e40af,stroke:#0b1f5e,color:#ffffff,stroke-width:2px
  classDef module fill:#2563eb,stroke:#0b1f5e,color:#ffffff,stroke-width:1.5px
  classDef internal fill:#60a5fa,stroke:#1e3a8a,color:#0b1020,stroke-width:1.2px
  classDef io fill:#fb923c,stroke:#9a3412,color:#1f1300,stroke-width:1.5px
  classDef model fill:#22c55e,stroke:#14532d,color:#06210f,stroke-width:1.5px
  classDef runtime fill:#9ca3af,stroke:#374151,color:#111827,stroke-dasharray: 6 3,stroke-width:1.2px
  classDef external fill:#f3f4f6,stroke:#6b7280,color:#111827,stroke-width:1.2px
  classDef artifact fill:#fde68a,stroke:#92400e,color:#1f1300,stroke-dasharray: 3 3,stroke-width:1px
  classDef note fill:#fef3c7,stroke:#b45309,color:#1f1300,stroke-width:1px
```
