How tranxuantuyen/vosagent works
Project Type and Purpose
• This is a Python-based multi-agent LLM (large language model) orchestration pipeline.
• Its goal is to take raw inputs (e.g., images, narratives), run a series of extraction and filtering steps, invoke one or more LLMs via custom prompts, then consolidate and output refined queries or narratives.
Top-Level Directory Analysis
• Core modules:
– agent.py, agent_lib.py, agent_debug.py: the central “agent” framework and helper code.
– config.py: global settings (model endpoints, API keys, thresholds).
– tools.py, utils.py: shared utilities (I/O, logging).
• Pipeline scripts (leaf-nodes of the DAG):
– extract_narrative.py, extract_object_name.py, extract_same_query.py, filter_object.py, merge_with_base.py, consolidate_query.py, segmentation.py, visulize.py
– multi_agent.py: orchestrates multiple agents in sequence or in parallel.
• “custom_prompts/”: YAML-encoded prompt templates per agent.
• “processed_data/”: serialized JSON outputs at various stages (narratives, object names, subsets).
• “script/”: glue scripts for external model servers (ollama, qwen API, slurm job submission, VQA).
Key Architectural Patterns
• Pipeline (ETL-style) of discrete Python modules, each performing one transformation.
• Agent orchestration layer that:
1) loads data from processed_data or raw inputs,
2) selects prompt,
3) calls external LLM service (through script wrappers),
4) writes back to processed_data.
• External services (Ollama server, QWEN API, VQA engine) are invoked via thin adapters in “script/.”
Main Components
A. Input Layer
• Raw data (images, text, JSON metadata).
• “segmentation.py” for vision preprocessing if images are used.
B. Pre-Processing Modules
• extract_narrative, extract_object_name, extract_same_query, filter_object
• Each module reads previous stage’s JSON and outputs new JSON.
C. Agent Core
• agent_lib.py (core agent behaviors) + agent.py (entry point)
• custom_prompts/ (YAML templates)
• config.py (connects to LLM endpoints)
D. External LLM Services
• Ollama (local server)
• QWEN API (remote, including 32GB variant)
• VQA engine (via slurm or shell scripts)
E. Orchestration Layer
• multi_agent.py (workflow manager)
• consolidate_query.py (final aggregator)
F. Data Store
• processed_data/ JSON files
G. Visualization / Output
• visulize.py (renders final outputs)
Relationships and Data Flow
• Orchestration Layer mediates the sequence and branching (multi_agent.py calls each module in turn)
• Consolidation module reads from multiple JSON files and writes final queries
Technology / Framework Call-outs
• Python 3.x
• YAML prompt files
• HTTP/RPC adapters to Ollama, QWEN
• JSON as intermediate format
• Shell/Slurm for batch VQA
Essential Elements
– Clear labels (module names, data files, services)
• Python code modules
• Data stores
• External services
– Optional: version or prototype modules (draft/) grayed out as “experimental.”
• how raw inputs traverse a chain of Python scripts
• how the agent framework invokes external LLM services via custom prompts
• how outputs are consolidated and stored
ensuring every critical component and interaction is captured accurately.