# zzzzchs/chat-scene architecture

> Architecture diagram of the GitHub repository zzzzchs/chat-scene, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zzzzchs/chat-scene
- Repository: https://github.com/zzzzchs/chat-scene

## Overview

##### Project Type and Purpose
- Chat-Scene is a research codebase for a multi-modal large language model specialized in 3D scene understanding (grounding, captioning, QA).
- It is not a web or mobile app, but a machine-learning “pipeline” project with distinct preprocessing, dataset construction, model training/inference, and evaluation stages.

##### Main Architectural Components
a. Data Preprocessing Layer
- Location: folder `preprocess/`, plus shell driver `preprocess/run_prepare.sh`.
- Responsibilities: ingest raw 3D scans (ScanNet, ScanRefer, etc.), extract/encode features (Mask3D outputs, DINOv2 image embeddings), generate annotation JSONs for each downstream task.
- Output: standardized dataset files (point clouds, image token embeddings, annotation JSONs) saved under `dataset/` or a specified data directory.

b. Dataset & DataLoader Layer
- Location: `dataset/` (modules: `base_dataset.py`, `dataloader.py`, `dataset_train.py`, `dataset_val.py`).
- Responsibilities: define PyTorch `Dataset` subclasses for each training/validation split and instantiate `DataLoader`s to batch multi-modal samples.
- Interfaces to Preprocessing outputs and hands off mini-batches of tokens, point cloud data, object IDs, prompt templates.

c. Model Core Layer
- Location: `models/` and subpackages (`chat3d.py`, `configuration_llama.py`, `modeling_llama.py`, `moe/`, `transformer_vanilla/`).
- Subcomponents:
- LLM Backbone: Vicuna-7B (via HuggingFace), loaded and LoRA-tuned (`load_llama.py`, `moe/moe_lora.py`).
- 3D Fusion & Embedding Modules: point-cloud position embeddings (`position_embedding.py`), object-identifier layers, mixture-of-experts attention (`moe/layer.py`, `transformer_vanilla/self_attention.py`).
- Chat-Scene Wrapper: `chat3d.py` glues the LLM, embeddings, and prompts for forward passes.
- Responsibilities: define forward, loss calculation, and task-specific heads (grounding, captioning, QA).

d. Training & Inference Orchestration
- Location: `tasks/train.py` for the unified training loop, plus `scripts/run.sh` (with `config.py`) as the entrypoint.
- Responsibilities: parse command-line args (`train_tag`, `val_tag`, `evaluate`), initialize distributed training (via `utils/distributed.py`), instantiate datasets, model, optimizer/scheduler (`utils/optimizer.py`, `utils/scheduler.py`), launch training/evaluation epochs, log metrics (`utils/logger.py`).

e. Evaluation & Visualization
- Responsibilities: post-hoc metric computation (BLEU, CIDEr via `utils/capeval/`), generate plots and qualitative visuals.

f. Utilities & Configuration
- Location: `utils/` (common functions, config loaders, distributed helpers, schedulers, evaluators).
- Responsibilities: reusable code—config parsing (`config_utils.py`), logging, metric wrappers, helper functions.

g. Prompts & Assets
- Location: `prompts/` and `assets/`.
- Responsibilities: store GPT-style prompt templates, image assets for README, benchmark charts.

h. External Dependencies & Resources
- HuggingFace model hub (Vicuna weights)
- PyTorch, NVIDIA CUDA, mixed‐precision training libraries
- 3D dataset repositories (ScanNet, ScanRefer, etc.)

##### Interactions and Data Flow
1. Raw data + scans → Preprocessing Layer → Serialized dataset files
2. Dataset Layer reads files → produces batched samples
3. Batches → Model Core forward pass (LLM + 3D embeddings)
4. Loss → backprop → optimizer/scheduler → model update
5. Periodic checkpoints → Evaluation & Visualization → Metrics & Plots
6. Prompts fed into Model Core at inference time

##### Architectural Patterns & Principles
- Pipeline pattern: clear stage separation (preprocess → dataset → model → training → evaluation).
- Modular design: each folder is a plug-and-play component (e.g., swap in a different detector in preprocess, change LLM in models).
- Config-driven execution: `scripts/config.py` + argument parsing control behavior without code changes.
- Reuse & Extensibility: `utils/` and `others/` hold generic helpers and custom scripts, allowing easy extension to new tasks.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data and Services
    subgraph "External Resources"
        direction TB
        HF["HuggingFace Model Hub"]:::external
        SN["ScanNet, ScanRefer Repos"]:::external
    end

    %% Preprocessing Layer
    subgraph "Data Preprocessing Layer"
        direction TB
        PREP_FOLDER["preprocess/"]:::preprocess
        PREP_SCRIPT["run_prepare.sh"]:::preprocess
        DATASTORE1[(Processed Dataset Files)]:::storage
    end

    %% Dataset Layer
    subgraph "Dataset & DataLoader Layer"
        direction TB
        BASE_DS["base_dataset.py"]:::dataset
        DATALOADER["dataloader.py"]:::dataset
        DS_TRAIN["dataset_train.py"]:::dataset
        DS_VAL["dataset_val.py"]:::dataset
    end

    %% Model Core Layer
    subgraph "Model Core Layer"
        direction TB
        CHAT3D["chat3d.py"]:::model
        CONF_LLAMA["configuration_llama.py"]:::model
        LOAD_LLAMA["load_llama.py"]:::model
        MODEL_LLAMA["modeling_llama.py"]:::model
        POS_EMB["position_embedding.py"]:::model
        subgraph "MoE Modules"
            direction TB
            MOE_LAYER["moe/layer.py"]:::model
            MOE_LORA["moe/moe_lora.py"]:::model
        end
        subgraph "Transformer Vanilla"
            direction TB
            SA["self_attention.py"]:::model
            MHSA["mhsa.py"]:::model
            T_BLOCK["transformer_block.py"]:::model
        end
        CHECKPOINT[(Model Checkpoints)]:::storage
    end

    %% Orchestration
    subgraph "Training & Inference Orchestration"
        direction TB
        TRAIN_TASK["tasks/train.py"]:::orchestration
        RUN_SCRIPT["scripts/run.sh"]:::orchestration
        CONFIG_SCRIPT["scripts/config.py"]:::orchestration
    end

    %% Evaluation & Visualization
    subgraph "Evaluation & Visualization"
        direction TB
        EVAL_OFF["eval_offline.py"]:::evaluation
        CALC_ACC["calc_scanrefer_grounding_acc.py"]:::evaluation
        GROUND_VIZ["ground_visualize.py"]:::evaluation
        VISUALIZE["visualize.py"]:::evaluation
        VIZ["viz.py"]:::evaluation
        METRICS[(Metrics & Plots)]:::storage
    end

    %% Utilities & Configuration
    subgraph "Utilities & Configuration"
        direction TB
        UTIL_CONF["config.py"]:::utils
        UTIL_CUTIL["config_utils.py"]:::utils
        UTIL_BUTIL["basic_utils.py"]:::utils
        UTIL_HELP["helper.py"]:::utils
        UTIL_BOX["box_utils.py"]:::utils
        UTIL_PC["pc_util.py"]:::utils
        UTIL_EVAL["eval.py"]:::utils
        UTIL_LOGGER["logger.py"]:::utils
        UTIL_DIST["distributed.py"]:::utils
    end

    %% Prompts & Assets
    subgraph "Prompts & Assets"
        direction TB
        PROMPTS["prompts/"]:::assets
        ASSETS["assets/"]:::assets
    end

    %% Data Flow and Control
    HF -->|Provides Vicuna Weights| LOAD_LLAMA
    SN -->|Raw Scans| PREP_FOLDER
    PREP_FOLDER --> PREP_SCRIPT
    PREP_SCRIPT -->|Generates| DATASTORE1
    DATASTORE1 -->|Reads Files| BASE_DS
    BASE_DS --> DATALOADER
    DATALOADER --> DS_TRAIN
    DATALOADER --> DS_VAL
    DS_TRAIN -->|Batches| CHAT3D
    DS_VAL -->|Batches| CHAT3D
    PROMPTS -->|Prompt Templates| CHAT3D
    CHAT3D -->|Forward Pass| MODEL_LLAMA
    MODEL_LLAMA --> POS_EMB
    POS_EMB --> MOE_LAYER
    MOE_LAYER --> MOE_LORA
    MODEL_LLAMA --> SA
    SA --> MHSA
    MHSA --> T_BLOCK
    TRAIN_TASK -->|Init Training| DATALOADER
    TRAIN_TASK -->|Loads Model| LOAD_LLAMA
    TRAIN_TASK -->|Updates| CHECKPOINT
    RUN_SCRIPT --> TRAIN_TASK
    CONFIG_SCRIPT --> RUN_SCRIPT
    UTIL_DIST --> TRAIN_TASK
    UTIL_CONF --> RUN_SCRIPT
    UTIL_CUTIL --> UTIL_CONF
    UTIL_LOGGER --> TRAIN_TASK
    UTIL_PC --> TRAIN_TASK
    CHECKPOINT -->|For Evaluation| EVAL_OFF
    DATASTORE1 -->|For Evaluation| EVAL_OFF
    EVAL_OFF --> CALC_ACC
    EVAL_OFF --> GROUND_VIZ
    EVAL_OFF --> VISUALIZE
    EVAL_OFF --> VIZ
    GROUND_VIZ --> METRICS
    VISUALIZE --> METRICS
    VIZ --> METRICS

    %% Click Events
    click PREP_FOLDER "https://github.com/zzzzchs/chat-scene/tree/dev/preprocess/"
    click PREP_SCRIPT "https://github.com/zzzzchs/chat-scene/blob/dev/preprocess/run_prepare.sh"
    click BASE_DS "https://github.com/zzzzchs/chat-scene/blob/dev/dataset/base_dataset.py"
    click DATALOADER "https://github.com/zzzzchs/chat-scene/blob/dev/dataset/dataloader.py"
    click DS_TRAIN "https://github.com/zzzzchs/chat-scene/blob/dev/dataset/dataset_train.py"
    click DS_VAL "https://github.com/zzzzchs/chat-scene/blob/dev/dataset/dataset_val.py"
    click CHAT3D "https://github.com/zzzzchs/chat-scene/blob/dev/models/chat3d.py"
    click CONF_LLAMA "https://github.com/zzzzchs/chat-scene/blob/dev/models/configuration_llama.py"
    click LOAD_LLAMA "https://github.com/zzzzchs/chat-scene/blob/dev/models/load_llama.py"
    click MODEL_LLAMA "https://github.com/zzzzchs/chat-scene/blob/dev/models/modeling_llama.py"
    click POS_EMB "https://github.com/zzzzchs/chat-scene/blob/dev/models/position_embedding.py"
    click MOE_LAYER "https://github.com/zzzzchs/chat-scene/blob/dev/models/moe/layer.py"
    click MOE_LORA "https://github.com/zzzzchs/chat-scene/blob/dev/models/moe/moe_lora.py"
    click SA "https://github.com/zzzzchs/chat-scene/blob/dev/models/transformer_vanilla/self_attention.py"
    click MHSA "https://github.com/zzzzchs/chat-scene/blob/dev/models/transformer_vanilla/mhsa.py"
    click T_BLOCK "https://github.com/zzzzchs/chat-scene/blob/dev/models/transformer_vanilla/transformer_block.py"
    click TRAIN_TASK "https://github.com/zzzzchs/chat-scene/blob/dev/tasks/train.py"
    click RUN_SCRIPT "https://github.com/zzzzchs/chat-scene/blob/dev/scripts/run.sh"
    click CONFIG_SCRIPT "https://github.com/zzzzchs/chat-scene/blob/dev/scripts/config.py"
    click UTIL_DIST "https://github.com/zzzzchs/chat-scene/blob/dev/utils/distributed.py"
    click UTIL_CONF "https://github.com/zzzzchs/chat-scene/blob/dev/utils/config.py"
    click UTIL_CUTIL "https://github.com/zzzzchs/chat-scene/blob/dev/utils/config_utils.py"
    click UTIL_BUTIL "https://github.com/zzzzchs/chat-scene/blob/dev/utils/basic_utils.py"
    click UTIL_HELP "https://github.com/zzzzchs/chat-scene/blob/dev/utils/helper.py"
    click UTIL_BOX "https://github.com/zzzzchs/chat-scene/blob/dev/utils/box_utils.py"
    click UTIL_PC "https://github.com/zzzzchs/chat-scene/blob/dev/utils/pc_util.py"
    click UTIL_EVAL "https://github.com/zzzzchs/chat-scene/blob/dev/utils/eval.py"
    click UTIL_LOGGER "https://github.com/zzzzchs/chat-scene/blob/dev/utils/logger.py"
    click CALC_ACC "https://github.com/zzzzchs/chat-scene/blob/dev/others/calc_scanrefer_grounding_acc.py"
    click EVAL_OFF "https://github.com/zzzzchs/chat-scene/blob/dev/others/eval_offline.py"
    click GROUND_VIZ "https://github.com/zzzzchs/chat-scene/blob/dev/others/ground_visualize.py"
    click VISUALIZE "https://github.com/zzzzchs/chat-scene/blob/dev/others/visualize.py"
    click VIZ "https://github.com/zzzzchs/chat-scene/blob/dev/others/viz.py"
    click PROMPTS "https://github.com/zzzzchs/chat-scene/tree/dev/prompts/"
    click ASSETS "https://github.com/zzzzchs/chat-scene/tree/dev/assets/"

    %% Styling
    classDef preprocess fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef dataset fill:#90EE90,stroke:#333,stroke-width:1px
    classDef model fill:#FFA500,stroke:#333,stroke-width:1px
    classDef orchestration fill:#DDA0DD,stroke:#333,stroke-width:1px
    classDef utils fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef assets fill:#F5DEB3,stroke:#333,stroke-width:1px
    classDef external fill:#ffffff,stroke:#333,stroke-dasharray: 5 5
    classDef evaluation fill:#FF6347,stroke:#333,stroke-width:1px
    classDef storage fill:#CCCCFF,stroke:#333,stroke-width:1px
```
