How zzzzchs/chat-scene works
Project Type and Purpose
• Chat-Scene is a research codebase for a multi-modal large language model specialized in 3D scene understanding (grounding, captioning, QA).
• It is not a web or mobile app, but a machine-learning “pipeline” project with distinct preprocessing, dataset construction, model training/inference, and evaluation stages.
Main Architectural Components
a. Data Preprocessing Layer
• Location: folder preprocess/, plus shell driver preprocess/run_prepare.sh.
• Responsibilities: ingest raw 3D scans (ScanNet, ScanRefer, etc.), extract/encode features (Mask3D outputs, DINOv2 image embeddings), generate annotation JSONs for each downstream task.
• Output: standardized dataset files (point clouds, image token embeddings, annotation JSONs) saved under dataset/ or a specified data directory.
b. Dataset & DataLoader Layer
• Location: dataset/ (modules: base_dataset.py, dataloader.py, dataset_train.py, dataset_val.py).
• Responsibilities: define PyTorch Dataset subclasses for each training/validation split and instantiate DataLoaders to batch multi-modal samples.
• Interfaces to Preprocessing outputs and hands off mini-batches of tokens, point cloud data, object IDs, prompt templates.
c. Model Core Layer
• Location: models/ and subpackages (chat3d.py, configuration_llama.py, modeling_llama.py, moe/, transformer_vanilla/).
• Subcomponents:
• LLM Backbone: Vicuna-7B (via HuggingFace), loaded and LoRA-tuned (load_llama.py, moe/moe_lora.py).
• 3D Fusion & Embedding Modules: point-cloud position embeddings (position_embedding.py), object-identifier layers, mixture-of-experts attention (moe/layer.py, transformer_vanilla/self_attention.py).
• Chat-Scene Wrapper: chat3d.py glues the LLM, embeddings, and prompts for forward passes.
• Responsibilities: define forward, loss calculation, and task-specific heads (grounding, captioning, QA).
d. Training & Inference Orchestration
• Location: tasks/train.py for the unified training loop, plus scripts/run.sh (with config.py) as the entrypoint.
• Responsibilities: parse command-line args (train_tag, val_tag, evaluate), initialize distributed training (via utils/distributed.py), instantiate datasets, model, optimizer/scheduler (utils/optimizer.py, utils/scheduler.py), launch training/evaluation epochs, log metrics (utils/logger.py).
e. Evaluation & Visualization
• Responsibilities: post-hoc metric computation (BLEU, CIDEr via utils/capeval/), generate plots and qualitative visuals.
f. Utilities & Configuration
• Location: utils/ (common functions, config loaders, distributed helpers, schedulers, evaluators).
• Responsibilities: reusable code—config parsing (config_utils.py), logging, metric wrappers, helper functions.
g. Prompts & Assets
• Location: prompts/ and assets/.
• Responsibilities: store GPT-style prompt templates, image assets for README, benchmark charts.
h. External Dependencies & Resources
• HuggingFace model hub (Vicuna weights)
• PyTorch, NVIDIA CUDA, mixed‐precision training libraries
• 3D dataset repositories (ScanNet, ScanRefer, etc.)
Interactions and Data Flow
1. Raw data + scans → Preprocessing Layer → Serialized dataset files
2. Dataset Layer reads files → produces batched samples
3. Batches → Model Core forward pass (LLM + 3D embeddings)
4. Loss → backprop → optimizer/scheduler → model update
5. Periodic checkpoints → Evaluation & Visualization → Metrics & Plots
6. Prompts fed into Model Core at inference time
Architectural Patterns & Principles
• Pipeline pattern: clear stage separation (preprocess → dataset → model → training → evaluation).
• Modular design: each folder is a plug-and-play component (e.g., swap in a different detector in preprocess, change LLM in models).
• Config-driven execution: scripts/config.py + argument parsing control behavior without code changes.
• Reuse & Extensibility: utils/ and others/ hold generic helpers and custom scripts, allowing easy extension to new tasks.