# controlnet/hydra architecture

> Architecture diagram of the GitHub repository controlnet/hydra, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/controlnet/hydra
- Repository: https://github.com/controlnet/hydra

## Overview

##### Project Type & Purpose
- This is a multimodal (vision and language) deep learning system that implements a “Hyper Agent” which uses reinforcement learning to perform dynamic compositional visual reasoning.
- The README emphasizes the use of reinforcement learning (DQN) to fine–tune an LLM for tasks like image–based question answering, visual inference (e.g., OK-VQA, RefCOCO) and integration with tools from module_repos (GLIP, Grounded–Segment–Anything, LLaVA).

##### Core Components
a. User Interfaces:
– Command Line Interface (CLI): demo_cli.py
– Gradio–based GUI: demo_gradio.py
– Possibly other interactive demos (video demos in media, notebooks in repos).
b. Agent & Controller:
– The “hydra_vl4ai/agent” folder hosts essential modules:
- Controller, Hydra, and Planner for coordinating tasks.
- LLM integration (llm.py) which makes external API calls (OpenAI, ollama for Llama, etc.).
- Prompt management with subdirectories for different versions (ChatGPT, Llama–31, etc.) and various task adaptations (gqa, okvqa, refcoco).
c. Execution & Inference Modules:
– hydra_vl4ai/execution/ (image_patch.py, toolbox.py) to handle low–level image processing or other task–specific execution.
– hydra_vl4ai/executor.py for managing the overall inference worker.
d. Training & Reinforcement Learning:
– train.py contains the training loop to run RL (DQN) for the controller.
– Config files in the config folder (such as dqn_debug.yaml, dqn_train_config.yaml) specify hyperparameters for the RL component.
e. Evaluation:
– evaluate.py and hydra_vl4ai/evaluation/ (vqa_eval.py, grounding_eval.py) which are used to analyze system outputs.
f. Tools & External Integration:
– hydra_vl4ai/tool/ and model subdirectories that wrap functionality of external models (BLIP, CLIP, efficient_sam, glip, etc.)
– module_repos folder contains integrated repositories (GLIP, Grounded–Segment–Anything, LLaVA, etc.) which supplement the system with state–of–the–art models.
g. Supporting Infrastructure:
– Config management (files in the config folder, .env.example, build_env.sh) and environment setup.
– Utility modules (in hydra_vl4ai/util/) for configuration handling, console messaging, and logging.

##### Relationships and Data Flow
– The user (or automated pipeline) submits an image and prompt through the CLI or Gradio GUI.
– The front–end passes the request to the Hydra Agent controller which:
- Uses prompt modules (organized by target task, e.g., OK-VQA or RefCOCO) to format the query.
- Interacts with the LLM (via llm.py) that may call external APIs (OpenAI, ollama, etc.).
– The controller then relies on a planning module (planner.py) which may decompose the task into subtasks.
– For each subtask, execution modules (execution/, tool/) are invoked to perform visual or multimodal processing.
– In training, the train.py script uses RL via DQN (configured by dqn*.yaml) to update the decision-making policy of the controller (agent actions, state memory via hydra_vl4ai/agent/smb).
– The evaluation modules assess the output, with results saved and later processed (main.py and evaluate.py).

##### Architectural Patterns and Design Principles
- Modular Design: Each major functionality (interface, control, execution, evaluation) is isolated in its own sub–module.
- Separation of Concerns: Deep separation exists between the agent’s decision logic, the processing/execution chain, and external tool integrations.
- Plug–and–Play Extensibility: The use of (a) configuration files to toggle model parameters, (b) clearly demarcated prompt directories, and (c) external module repositories suggest that the system is designed for rapid prototyping and experimentation.
- Multimodal Fusion: Integration between vision models (GLIP, Grounded–Segment–Anything) and language modules (LLM APIs) is central.
- Reinforcement Learning Loop: Represent the training pipeline where the RL controller receives feedback from the environment (evaluations) and updates its policy.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Interface Layer
    subgraph "Interface Layer"
        cli["CLI Demo (demo_cli.py)"]:::ui
        gradio["Gradio GUI Demo (demo_gradio.py)"]:::ui
    end

    %% Control Layer (HYDRA Agent)
    subgraph "Control Layer"
        controller["Controller (hydra_vl4ai/agent/)"]:::agent
        hydra["Hydra (hydra_vl4ai/agent/)"]:::agent
        planner["Planner (hydra_vl4ai/agent/)"]:::agent
        llm["LLM Integration (hydra_vl4ai/agent/llm.py)"]:::agent
        prompt["Prompt Management (hydra_vl4ai/agent/prompt/)"]:::agent
        rldqn["RL Component (hydra_vl4ai/agent/rl_dqn.py)"]:::agent
        state["State Memory (hydra_vl4ai/agent/smb/state_memory_bank.py)"]:::agent
    end

    %% Execution & Inference Layer
    subgraph "Execution & Inference"
        execution["Image Processing & Toolbox (hydra_vl4ai/execution/)"]:::exec
        executor["Inference Manager (hydra_vl4ai/executor.py)"]:::exec
    end

    %% Training & Evaluation Layer
    subgraph "Training & Evaluation"
        train["RL Training Loop (train.py)"]:::eval
        configDQN["DQN Config Files (config/dqn_debug.yaml, config/dqn_train_config.yaml)"]:::eval
        evaluate["Evaluation Script (evaluate.py)"]:::eval
        evalmodules["Evaluation Modules (hydra_vl4ai/evaluation/)"]:::eval
    end

    %% Tools & External Integration
    subgraph "Tools & External Integration"
        tool["External Model Wrappers (hydra_vl4ai/tool/)"]:::ext
        modrepos["Integrated External Repos (module_repos/)"]:::ext
    end

    %% Supporting Infrastructure
    subgraph "Support Infrastructure"
        config["Configuration Management (config/)"]:::util
        util["Utility Modules (hydra_vl4ai/util/)"]:::util
    end

    %% Data Flow Connections
    cli -->|"input"| controller
    gradio -->|"input"| controller

    controller -->|"coordinates"| hydra
    controller -->|"delegates"| planner
    controller -->|"calls"| llm
    controller -->|"formats"| prompt
    controller -->|"triggers"| rldqn
    controller -->|"uses"| state

    planner -->|"decomposes tasks"| execution
    prompt -->|"supplies query"| execution
    llm -->|"fetches response"| execution

    execution -->|"manages processing"| executor
    executor -->|"provides output"| evaluate
    executor -->|"supplies data"| evalmodules

    evaluate -->|"feedback"| train
    evalmodules -->|"feedback"| train
    train -->|"updates policy"| rldqn
    train -->|"configured by"| configDQN

    controller -->|"integrates"| tool
    planner -->|"leverages"| tool
    execution -->|"augments"| tool
    tool -->|"enhances"| executor
    tool -->|"uses data from"| modrepos

    %% Support arrows (dotted for supporting infra)
    config --- controller
    config --- execution
    config --- train
    util --- controller
    util --- execution
    util --- train

    %% Styling
    classDef ui fill:#B3CDE0,stroke:#333,stroke-width:2px;
    classDef agent fill:#C1E1C1,stroke:#333,stroke-width:2px;
    classDef exec fill:#FAD7A0,stroke:#333,stroke-width:2px;
    classDef eval fill:#F1948A,stroke:#333,stroke-width:2px;
    classDef util fill:#D3D3D3,stroke:#333,stroke-width:2px;
    classDef ext fill:#D7BDE2,stroke:#333,stroke-width:2px;

    %% Click Events
    click cli "https://github.com/controlnet/hydra/blob/master/demo_cli.py"
    click gradio "https://github.com/controlnet/hydra/blob/master/demo_gradio.py"
    click controller "https://github.com/controlnet/hydra/tree/master/hydra_vl4ai/agent/"
    click hydra "https://github.com/controlnet/hydra/tree/master/hydra_vl4ai/agent/"
    click planner "https://github.com/controlnet/hydra/tree/master/hydra_vl4ai/agent/"
    click llm "https://github.com/controlnet/hydra/blob/master/hydra_vl4ai/agent/llm.py"
    click prompt "https://github.com/controlnet/hydra/tree/master/hydra_vl4ai/agent/prompt/"
    click rldqn "https://github.com/controlnet/hydra/blob/master/hydra_vl4ai/agent/rl_dqn.py"
    click state "https://github.com/controlnet/hydra/blob/master/hydra_vl4ai/agent/smb/state_memory_bank.py"
    click execution "https://github.com/controlnet/hydra/tree/master/hydra_vl4ai/execution/"
    click executor "https://github.com/controlnet/hydra/blob/master/hydra_vl4ai/executor.py"
    click train "https://github.com/controlnet/hydra/blob/master/train.py"
    click configDQN "https://github.com/controlnet/hydra/blob/master/config/dqn_debug.yaml"
    click evaluate "https://github.com/controlnet/hydra/blob/master/evaluate.py"
    click evalmodules "https://github.com/controlnet/hydra/tree/master/hydra_vl4ai/evaluation/"
    click tool "https://github.com/controlnet/hydra/tree/master/hydra_vl4ai/tool/"
    click modrepos "https://github.com/controlnet/hydra/tree/master/module_repos/"
    click config "https://github.com/controlnet/hydra/tree/master/config/"
    click util "https://github.com/controlnet/hydra/tree/master/hydra_vl4ai/util/"
```
