# megvii-research/far3d architecture

> Architecture diagram of the GitHub repository megvii-research/far3d, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/megvii-research/far3d
- Repository: https://github.com/megvii-research/far3d

## Overview

##### Project Type and Purpose
- Type: Research framework for 3D object detection on surrounding views (especially long-range objects) using a novel sparse query-based approach.
- Purpose: Integrate high-quality 2D object priors and construct adaptive 3D queries to efficiently learn multi-view, multi-scale features. The design relies on modules from MMDetection3D and extends them with additional pipelines (e.g., perspective-aware aggregation and range-modulated 3D denoising).

##### Analyzing the File Structure
- The core of the project is under “projects/mmdet3d_plugin” which includes:
- core (apis, bbox handling, evaluation procedures)
- datasets (data loaders, augmentation pipelines, samplers for multi-view 3D data)
- models (organized into subdirectories for backbones, dense heads, detectors, depth predictor, necks, and utilities)
- Additional directories (such as “tools” and “docs”) provide scripts for training, testing, analysis, and visualization.
- The configuration file (e.g., projects/configs/far3d.py) ties this all together and is the entry point for running experiments.

##### Examining the README
- The README explains that Far3D implements a novel sparse query-based design, leveraging high-quality 2D object priors to guide 3D adaptive queries.
- It highlights key modules (such as the perspective-aware aggregation module and 3D denoising) and notes that the pipeline is based on StreamPETR.
- It includes instructions for training and testing along with citations to related works and comparisons.

a. Main Components to Include:
- Configuration & Setup: projects/configs/far3d.py (defines experiment settings, training parameters, network architecture choices)
- Data Pipeline:
- Datasets: Modules under projects/mmdet3d_plugin/datasets that load and preprocess data (e.g., Argoverse2, nuScenes variants)
- Pipelines: Data augmentation, transformation steps (custom_pipeline, formatting, transform_3d)
- Samplers: Managing distributed sampling and grouping.
- Model Architecture:
- Backbones: e.g., VoVnet variants found in models/backbones for feature extraction.
- Necks: Feature pyramid networks (FPN) as seen in projects/mmdet3d_plugin/models/necks.
- Detectors & Dense Heads: Modules in models/detectors and models/dense_heads that handle object detection; here, “far3d.py” is visible along with other heads.
- Depth Predictor: Including modules for depth estimation (helpful for 3D localization).
- Utility Modules: Attention, positional encoding, transformer utilities within models/utils.
- Query Modules: Elements implemented for sparse, adaptive 3D queries (implied by the training APIs and detection heads).
- Training and Evaluation:
- API Scripts: Located in core/apis (train.py, test.py, mmdet_train.py) to manage model training, inference, and evaluations.
- Tools: A suite of scripts under “tools” for distributed training/testing, visualizations, and checkpoint handling.
- Visualization & Debugging:
- Tools under “tools/visual” which provide notebooks and scripts for 2D/3D visualizations (e.g., vis_2d, vis_3dpred).

b. Relationships and Interactions:
- Data flows from the dataset loaders (and pipelines) into the training APIs.
- The configuration file (far3d.py) centrally connects settings to both the model architecture and data pipeline.
- The model components (backbones, necks, detectors, etc.) interact sequentially – raw data ➜ feature extraction ➜ feature aggregation and transformation ➜ prediction (via dense heads/detectors).
- External scripts (tools) manage orchestration (e.g., distributed training, evaluation, visualization) around these core components.
- Utility modules inform various parts of the model with additional algorithms (for attention, transformer designs, etc.).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Main Components
    Config["Configuration & Setup"]:::config
    DP["Data Pipeline"]:::data
    MA["Model Architecture"]:::model
    TE["Training & Evaluation"]:::train
    VD["Visualization & Debugging"]:::vis

    %% Subgraph for Data Pipeline
    subgraph "Data Pipeline" [Data Pipeline]
        Datasets["Dataset Loaders"]:::data
        Pipelines["Transformation Pipelines"]:::data
        Samplers["Samplers"]:::data
    end

    %% Subgraph for Model Architecture
    subgraph "Model Architecture" [Model Architecture]
        Backbones["Backbones"]:::model
        Necks["Necks"]:::model
        Detectors["Detectors"]:::model
        DenseHeads["Dense Heads"]:::model
        DepthPred["Depth Predictor"]:::model
        Utils["Utility Modules"]:::model
        QueryMods["Query Modules"]:::model
    end

    %% Subgraph for Training & Evaluation
    subgraph "Training & Evaluation" [Training & Evaluation]
        TrainingAPIs["Training APIs"]:::train
        Orchestration["Orchestration Scripts"]:::train
    end

    %% Subgraph for Visualization & Debugging
    subgraph "Visualization & Debugging" [Visualization & Debugging]
        VisTools["Visualization Tools"]:::vis
    end

    %% Relationships from Configuration
    Config -->|"sets"| Datasets
    Config -->|"sets"| Pipelines
    Config -->|"sets"| Samplers
    Config -->|"configures"| Backbones
    Config -->|"configures"| Necks
    Config -->|"configures"| Detectors
    Config -->|"configures"| DenseHeads
    Config -->|"configures"| DepthPred
    Config -->|"configures"| Utils
    Config -->|"configures"| QueryMods
    Config -->|"inits"| TrainingAPIs

    %% Data Flow: Data Pipeline to Model Architecture
    Datasets -->|"feeds data"| Backbones
    Pipelines -->|"prepares"| Datasets
    Samplers -->|"organizes"| Datasets
    DP -->|"provides input"| MA

    %% Internal Model Architecture Flow
    Backbones -->|"extracts features"| Necks
    Necks -->|"aggregates"| Detectors
    Necks -->|"aggregates"| DenseHeads
    Necks -->|"aggregates"| DepthPred
    Necks -->|"aggregates"| Utils
    Necks -->|"guides"| QueryMods

    %% Connection to Training & Evaluation
    MA -->|"trains on"| TrainingAPIs
    TrainingAPIs -->|"orchestrated by"| Orchestration

    %% Connection to Visualization
    TrainingAPIs -->|"outputs to"| VisTools


    %% Click Events for Component Mappings
    click Config "https://github.com/megvii-research/far3d/blob/main/projects/configs/far3d.py"
    click Datasets "https://github.com/megvii-research/far3d/tree/main/projects/mmdet3d_plugin/datasets/"
    click Pipelines "https://github.com/megvii-research/far3d/tree/main/projects/mmdet3d_plugin/datasets/pipelines/"
    click Samplers "https://github.com/megvii-research/far3d/tree/main/projects/mmdet3d_plugin/datasets/samplers/"
    click Backbones "https://github.com/megvii-research/far3d/tree/main/projects/mmdet3d_plugin/models/backbones/"
    click Necks "https://github.com/megvii-research/far3d/tree/main/projects/mmdet3d_plugin/models/necks/"
    click Detectors "https://github.com/megvii-research/far3d/tree/main/projects/mmdet3d_plugin/models/detectors/"
    click DenseHeads "https://github.com/megvii-research/far3d/tree/main/projects/mmdet3d_plugin/models/dense_heads/"
    click DepthPred "https://github.com/megvii-research/far3d/tree/main/projects/mmdet3d_plugin/models/depth_predictor/"
    click Utils "https://github.com/megvii-research/far3d/tree/main/projects/mmdet3d_plugin/models/utils/"
    click TrainingAPIs "https://github.com/megvii-research/far3d/tree/main/projects/mmdet3d_plugin/core/apis/"
    click Orchestration "https://github.com/megvii-research/far3d/tree/main/tools/"
    click VisTools "https://github.com/megvii-research/far3d/tree/main/tools/visual/"

    %% Styles
    classDef config fill:#ffd700,stroke:#333,stroke-width:2px;
    classDef data fill:#87cefa,stroke:#333,stroke-width:2px;
    classDef model fill:#ffcccb,stroke:#333,stroke-width:2px;
    classDef train fill:#dda0dd,stroke:#333,stroke-width:2px;
    classDef vis fill:#ffdab9,stroke:#333,stroke-width:2px;
```
