# tranxuantuyen/vosagent architecture

> Architecture diagram of the GitHub repository tranxuantuyen/vosagent, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tranxuantuyen/vosagent
- Repository: https://github.com/tranxuantuyen/vosagent

## Overview

##### Project Type and Purpose
- This is a Python-based multi-agent LLM (large language model) orchestration pipeline.
- Its goal is to take raw inputs (e.g., images, narratives), run a series of extraction and filtering steps, invoke one or more LLMs via custom prompts, then consolidate and output refined queries or narratives.

##### Top-Level Directory Analysis
- Core modules:
– agent.py, agent_lib.py, agent_debug.py: the central “agent” framework and helper code.
– config.py: global settings (model endpoints, API keys, thresholds).
– tools.py, utils.py: shared utilities (I/O, logging).
- Pipeline scripts (leaf-nodes of the DAG):
– extract_narrative.py, extract_object_name.py, extract_same_query.py, filter_object.py, merge_with_base.py, consolidate_query.py, segmentation.py, visulize.py
– multi_agent.py: orchestrates multiple agents in sequence or in parallel.
- “custom_prompts/”: YAML-encoded prompt templates per agent.
- “processed_data/”: serialized JSON outputs at various stages (narratives, object names, subsets).
- “script/”: glue scripts for external model servers (ollama, qwen API, slurm job submission, VQA).

##### Key Architectural Patterns
- Pipeline (ETL-style) of discrete Python modules, each performing one transformation.
- Agent orchestration layer that:
1) loads data from processed_data or raw inputs,
2) selects prompt,
3) calls external LLM service (through script wrappers),
4) writes back to processed_data.
- External services (Ollama server, QWEN API, VQA engine) are invoked via thin adapters in “script/.”

##### Main Components
A. Input Layer
- Raw data (images, text, JSON metadata).
- “segmentation.py” for vision preprocessing if images are used.
B. Pre-Processing Modules
- extract_narrative, extract_object_name, extract_same_query, filter_object
- Each module reads previous stage’s JSON and outputs new JSON.
C. Agent Core
- agent_lib.py (core agent behaviors) + agent.py (entry point)
- custom_prompts/ (YAML templates)
- config.py (connects to LLM endpoints)
D. External LLM Services
- Ollama (local server)
- QWEN API (remote, including 32GB variant)
- VQA engine (via slurm or shell scripts)
E. Orchestration Layer
- multi_agent.py (workflow manager)
- consolidate_query.py (final aggregator)
F. Data Store
- processed_data/ JSON files
G. Visualization / Output
- visulize.py (renders final outputs)

##### Relationships and Data Flow
- Orchestration Layer mediates the sequence and branching (multi_agent.py calls each module in turn)
- Consolidation module reads from multiple JSON files and writes final queries

##### Technology / Framework Call-outs
- Python 3.x
- YAML prompt files
- HTTP/RPC adapters to Ollama, QWEN
- JSON as intermediate format
- Shell/Slurm for batch VQA

##### Essential Elements
– Clear labels (module names, data files, services)
- Python code modules
- Data stores
- External services
– Optional: version or prototype modules (draft/) grayed out as “experimental.”

- how raw inputs traverse a chain of Python scripts
- how the agent framework invokes external LLM services via custom prompts
- how outputs are consolidated and stored
ensuring every critical component and interaction is captured accurately.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Layer
    subgraph "Input Layer"
        direction TB
        RawData["Raw Data\n(images, text, metadata)"]:::external
        Segmentation["segmentation.py"]:::pipeline
    end

    %% Pre-Processing Modules
    subgraph "Pre-Processing" 
        direction TB
        ENarrative["extract_narrative.py"]:::pipeline
        EObjectName["extract_object_name.py"]:::pipeline
        ESameQuery["extract_same_query.py"]:::pipeline
        FilterObject["filter_object.py"]:::pipeline
        MergeBase["merge_with_base.py"]:::pipeline
    end

    %% Agent Framework
    subgraph "Agent Framework"
        direction TB
        AgentEntry["agent.py"]:::core
        AgentLib["agent_lib.py"]:::core
        Config["config.py"]:::core
        Prompts["custom_prompts/"]:::core
    end

    %% External Models
    subgraph "External Models"
        direction TB
        OllamaSrv["Ollama Server"]:::external
        QWEN["QWEN API"]:::external
        VQAEng["VQA Engine"]:::external
        subgraph "Adapters (script/)"
            direction TB
            OllamaAdapter["ollama_sever.py"]:::external
            QWEN32["qwen_api_32GB.py"]:::external
            QWENStd["qwen_api.py"]:::external
            RunOllama["run_ollama.sh"]:::external
            VQASlurm["vqa_slurm.sh"]:::external
            VQAScript["vqa_script_32GB.sh"]:::external
        end
    end

    %% Data Store
    DataStore["processed_data/"]:::data

    %% Orchestration Layer
    subgraph "Orchestration"
        direction TB
        MultiAgent["multi_agent.py"]:::orchestration
        Consolidate["consolidate_query.py"]:::orchestration
    end

    %% Visualization
    Visualize["visulize.py"]:::pipeline

    %% Data Flow Arrows
    RawData --> Segmentation
    Segmentation -->|writes JSON| DataStore

    DataStore --> ENarrative
    ENarrative -->|writes JSON| DataStore
    DataStore --> EObjectName
    EObjectName -->|writes JSON| DataStore
    DataStore --> ESameQuery
    ESameQuery -->|writes JSON| DataStore
    DataStore --> FilterObject
    FilterObject -->|writes JSON| DataStore
    DataStore --> MergeBase
    MergeBase -->|writes JSON| DataStore

    %% Agent Framework to External Models
    AgentEntry --> AgentLib
    AgentEntry --> Config
    AgentEntry --> Prompts
    AgentLib --> OllamaAdapter
    AgentLib --> QWENStd
    AgentLib --> QWEN32
    AgentLib --> RunOllama
    AgentLib --> VQASlurm
    AgentLib --> VQAScript

    OllamaAdapter --> OllamaSrv
    QWENStd --> QWEN
    QWEN32 --> QWEN
    RunOllama --> OllamaSrv
    VQASlurm --> VQAEng
    VQAScript --> VQAEng

    OllamaSrv -->|response JSON| DataStore
    QWEN -->|response JSON| DataStore
    VQAEng -->|response JSON| DataStore

    %% Orchestration Control
    MultiAgent -.-> Segmentation
    MultiAgent -.-> ENarrative
    MultiAgent -.-> EObjectName
    MultiAgent -.-> ESameQuery
    MultiAgent -.-> FilterObject
    MultiAgent -.-> MergeBase
    MultiAgent -.-> AgentEntry
    MultiAgent -.-> Consolidate

    %% Consolidation
    DataStore --> Consolidate
    Consolidate -->|writes final JSON| DataStore

    %% Visualization
    DataStore --> Visualize

    %% Click Events
    click Segmentation "https://github.com/tranxuantuyen/vosagent/blob/main/segmentation.py"
    click ENarrative "https://github.com/tranxuantuyen/vosagent/blob/main/extract_narrative.py"
    click EObjectName "https://github.com/tranxuantuyen/vosagent/blob/main/extract_object_name.py"
    click ESameQuery "https://github.com/tranxuantuyen/vosagent/blob/main/extract_same_query.py"
    click FilterObject "https://github.com/tranxuantuyen/vosagent/blob/main/filter_object.py"
    click MergeBase "https://github.com/tranxuantuyen/vosagent/blob/main/merge_with_base.py"
    click AgentEntry "https://github.com/tranxuantuyen/vosagent/blob/main/agent.py"
    click AgentLib "https://github.com/tranxuantuyen/vosagent/blob/main/agent_lib.py"
    click Config "https://github.com/tranxuantuyen/vosagent/blob/main/config.py"
    click Prompts "https://github.com/tranxuantuyen/vosagent/tree/main/custom_prompts/"
    click OllamaAdapter "https://github.com/tranxuantuyen/vosagent/blob/main/script/ollama_sever.py"
    click QWENStd "https://github.com/tranxuantuyen/vosagent/blob/main/script/qwen_api.py"
    click QWEN32 "https://github.com/tranxuantuyen/vosagent/blob/main/script/qwen_api_32GB.py"
    click RunOllama "https://github.com/tranxuantuyen/vosagent/blob/main/script/run_ollama.sh"
    click VQASlurm "https://github.com/tranxuantuyen/vosagent/blob/main/script/vqa_slurm.sh"
    click VQAScript "https://github.com/tranxuantuyen/vosagent/blob/main/script/vqa_script_32GB.sh"
    click MultiAgent "https://github.com/tranxuantuyen/vosagent/blob/main/multi_agent.py"
    click Consolidate "https://github.com/tranxuantuyen/vosagent/blob/main/consolidate_query.py"
    click DataStore "https://github.com/tranxuantuyen/vosagent/tree/main/processed_data/"
    click Visualize "https://github.com/tranxuantuyen/vosagent/blob/main/visulize.py"

    %% Styles
    classDef core fill:#ADD8E6,stroke:#0000FF,color:#000
    classDef pipeline fill:#90EE90,stroke:#008000,color:#000
    classDef orchestration fill:#FFA500,stroke:#FF8C00,color:#000
    classDef data fill:#FFFFE0,stroke:#FFD700,color:#000
    classDef external fill:#D3D3D3,stroke:#A9A9A9,color:#000
```
