# gengzezhou/navgpt architecture

> Architecture diagram of the GitHub repository gengzezhou/navgpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gengzezhou/navgpt
- Repository: https://github.com/gengzezhou/navgpt

## Overview

##### Project Type and Purpose
- NavGPT is a research-oriented agent for vision-and-language navigation, integrating advanced LLMs for explicit reasoning.
- The project focuses on sequential action prediction using LLMs, data processing for R2R (Room-to-Room) navigation, and custom LLM module integration.
- The README and file structure highlight key tasks: data processing, LLM interaction, navigation planning, and evaluation.

##### Analyzing the File Structure – Key Components
- Top-Level Structure:
- The root contains configuration files (.gitignore, LICENSE, README.md, and requirements.txt) and data folders (assets, datasets).
- Source Code (nav_src Directory):
- Core Navigation Agent: Files like NavGPT.py, agent.py, agent_base.py are the core logic runners.
- Environment and Data: ‘env.py’ defines the simulation or environment interactions. ‘data_utils.py’ helps in data handling.
- Evaluation: ‘eval_utils.py’ supports the validation and evaluation routines.
- LLM Integration: The “LLMs” subdirectory contains custom model integration (with an example file like Langchain_llama.py and documentation in Add_Custom_Models.md).
- Prompt Management: The “prompt” subdirectory holds the prompt templates and instruction generators (e.g., planner_prompt.py).
- Scripts: Under “scripts”, individual utility scripts (action_planner.py, merge_preds.py, etc.) handle tasks like preprocessing and result merging.
- Shared Utilities: The “utils” folder includes helper modules like logger, distributed computing, graph utilities, etc.

To capture the full architecture, consider including these essential elements:
a. Data & Input Components:
- Datasets Folder: Represents source data (navigation data, R2R datasets). Connect this to the data preprocessing scripts.
- Data Preprocessing: Modules found under nav_src/scripts that prepare and cleanse data.
b. Core Navigation Agent:
- Agent Module: Represent nav_src/agent.py and agent_base.py that encapsulate navigation logic.
- Environment Interaction: The env.py module depicts how the agent interacts with the simulated or real navigation environment.
c. LLM Integration:
- LLM Modules: Under nav_src/LLMs, show how the base LLM (e.g., GPT-3.5, GPT-4) or custom models are integrated. Indicate how the agent accesses LLM inference either via APIs (i.e., OpenAI API) or local modules.
- Prompt Templates: Illustrate the “prompt” folder as a source of instruction generation; these templates guide the reasoning process.
d. Evaluation & Logging:
- Evaluation Utilities: Include eval_utils.py which helps in the assessment of the generated navigation paths.
- Logging & Monitoring: Show the logger component (nav_src/utils/logger.py) to monitor agent actions and debugging information.
e. External Dependencies:
- Dependencies and Libraries: Mention dependency on Langchain (as reflected in the badges) and other libraries installed via requirements.txt.

##### Representing Interactions and Architectural Patterns
- From datasets to preprocessing scripts → fed into the environment and agent.
- From the agent and the prompt generator to the LLM modules (and then to external OpenAI APIs/custom LLMs), illustrating request and response cycles.
- From the agent back to evaluation/logging modules for performance tracking.
- Interactions between the planning module (prompt) and core agent logic ensuring explicit reasoning steps.
- Decoupling between components:
- Modular design where the data utilities, agent core, prompt engine, and LLM interfaces function as loosely coupled components.

- For each component, include:
- A label (e.g., “Agent Logic”, “Data Utils”, “LLM Integration”).
- Brief notes if needed (e.g., “Handles API calls”, “Manages prompt templates”).
- That the separation between core navigation functionalities (agent and environment) and support utilities (data, logging, evaluation) is clearly visible.
- Optionally, annotate the planning and reasoning steps central to the project’s approach (i.e., decomposing instructions, sub-goal generation).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data & Preprocessing"
        DS["Datasets Folder"]:::data
        DU["Data Utils"]:::util
        AP["action_planner.py"]:::script
        MP["merge_preds.py"]:::script
        DS --> DU
        DU --> AP
        DU --> MP
    end

    subgraph "Environment Layer"
        ENV["env.py"]:::env
        AP --> ENV
        MP --> ENV
    end

    subgraph "Core Navigation Agent"
        NG["NavGPT.py"]:::agent
        AG["agent.py"]:::agent
        AB["agent_base.py"]:::agent
        ENV --> NG
        ENV --> AG
        ENV --> AB
    end

    subgraph "LLM Integration & Prompting"
        LLM["LLMs Integration"]:::llm
        LC["Langchain_llama.py"]:::llm
        ACD["Add_Custom_Models.md"]:::llm
        PT["planner_prompt.py"]:::prompt
        NG --> LLM
        AG --> LLM
        AB --> LLM
        PT --> LLM
        LLM --> EX
    end

    subgraph "Evaluation & Logging"
        EV["eval_utils.py"]:::eval
        LG["logger.py"]:::log
        NG --> EV
        AG --> EV
        AB --> EV
        NG --> LG
        AG --> LG
        AB --> LG
    end

    EX["OpenAI API"]:::external

    %% Click Events
    click DS "https://github.com/gengzezhou/navgpt/tree/master/datasets"
    click DU "https://github.com/gengzezhou/navgpt/blob/master/nav_src/data_utils.py"
    click AP "https://github.com/gengzezhou/navgpt/blob/master/nav_src/scripts/action_planner.py"
    click MP "https://github.com/gengzezhou/navgpt/blob/master/nav_src/scripts/merge_preds.py"
    click ENV "https://github.com/gengzezhou/navgpt/blob/master/nav_src/env.py"
    click NG "https://github.com/gengzezhou/navgpt/blob/master/nav_src/NavGPT.py"
    click AG "https://github.com/gengzezhou/navgpt/blob/master/nav_src/agent.py"
    click AB "https://github.com/gengzezhou/navgpt/blob/master/nav_src/agent_base.py"
    click LLM "https://github.com/gengzezhou/navgpt/tree/master/nav_src/LLMs"
    click LC "https://github.com/gengzezhou/navgpt/blob/master/nav_src/LLMs/Langchain_llama.py"
    click ACD "https://github.com/gengzezhou/navgpt/blob/master/nav_src/LLMs/Add_Custom_Models.md"
    click PT "https://github.com/gengzezhou/navgpt/blob/master/nav_src/prompt/planner_prompt.py"
    click EV "https://github.com/gengzezhou/navgpt/blob/master/nav_src/eval_utils.py"
    click LG "https://github.com/gengzezhou/navgpt/blob/master/nav_src/utils/logger.py"

    %% Styles
    classDef agent fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef data fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef script fill:#FFFF99,stroke:#000,stroke-width:1px;
    classDef env fill:#FFDAB9,stroke:#000,stroke-width:1px;
    classDef llm fill:#DDA0DD,stroke:#000,stroke-width:1px;
    classDef prompt fill:#FFB6C1,stroke:#000,stroke-width:1px;
    classDef eval fill:#F0E68C,stroke:#000,stroke-width:1px;
    classDef log fill:#E0FFFF,stroke:#000,stroke-width:1px;
    classDef external fill:#FFC0CB,stroke:#000,stroke-width:1px;
    classDef util fill:#F5DEB3,stroke:#000,stroke-width:1px;
```
