# facebookresearch/convnext architecture

> Architecture diagram of the GitHub repository facebookresearch/convnext, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebookresearch/convnext
- Repository: https://github.com/facebookresearch/convnext

## Overview

##### Project Type and Purpose
- This is a research‐grade PyTorch codebase implementing ConvNeXt for:
a. Image classification on ImageNet (main pipeline)
b. Downstream object detection (using MMDetection)
c. Downstream semantic segmentation (using MMSegmentation)
- It provides training, evaluation, pre‐trained weights, and config‐driven workflows for all three tasks.

##### Core Modules & File‐Structure Analysis
- Root Scripts:
- main.py – entry point for classification training/eval
- engine.py – training/evaluation loop (losses, metrics, logging)
- datasets.py – dataset definitions & transformations (ImageNet, CIFAR, etc.)
- optim_factory.py – constructs optimizer & LR schedulers
- run_with_submitit.py – wrapper for distributed/slurm jobs
- utils.py – misc helpers (logging, checkpointing, seed setting)
- Model Definitions:
- models/convnext.py – core ConvNeXt blocks & variants (Tiny/Small/Base/Large)
- models/convnext_isotropic.py – isotropic variant
- Object Detection: object_detection/
- configs/… – MMDetection config hierarchy (base → ConvNeXt‐backboned Mask R‑CNN/Cascade Mask R‑CNN)
- mmcv_custom/… – custom hooks (text logging), optimizer constructors, checkpoint runners
- mmdet/… – vendored or extended MMDetection backbone definitions
- Semantic Segmentation: semantic_segmentation/
- configs/… – MMSegmentation config hierarchy (base → UPerNet w/ ConvNeXt backbones)
- mmcv_custom/… – custom Apex runner, text logging, optimizer constructors
- backbone/convnext.py – segmentation‐specific ConvNeXt modifications

##### High‐Level System Design Components
A. Entry Point (main.py or task‐specific scripts)
– Reads CLI args (model, data path, distributed setup)
– Loads configuration (via argument parsing + Python config files)
– Initializes logging, seeds, device (CPU/GPU/Distributed)
– Dispatches to classification or downstream task pipeline

B. Data Layer (datasets.py + config files)
– Defines DataLoader, data augmentations, dataset splits
– Task‐specific data readers (ImageNet, COCO, ADE20K) via config

C. Model Layer
– Core ConvNeXt architecture in models/
– Task‐specific “backbone” wrappers (in object_detection/mmdet and semantic_segmentation/backbone)
– Heads: classification head, detection heads (RPN, ROI), segmentation head

D. Training Engine (engine.py)
– Single‐/multi‐GPU training loop: forward, loss, backward, optimizer step, logging
– Evaluation loop: validation metrics, checkpoint saving

E. Optimization & Scheduling (optim_factory.py)
– Builds AdamW/Layer‐decay optimizers for ConvNeXt blocks
– Learning‐rate schedulers (cosine, step) as defined in config

F. Distributed & Cluster Execution (run_with_submitit.py)
– Wraps main.py to launch jobs on Slurm or multi‐node via Submitit
– Handles checkpoint resume & environment setup

G. Utilities (utils.py + mmcv_custom)
– Logging, checkpoint abstractions, custom checkpoint runners, text output, mixed‐precision support

##### Inter‐Component Relationships & Data Flow
- main.py → parses args → loads config → calls “run_pipeline”
- run_pipeline → calls dataset builder → model builder → engine.train/evaluate
- engine ← uses: model, optimizer (optim_factory), dataloader
- engine → logs via utils and optionally mmcv_custom runner → checkpoint via utils
- For downstream tasks: main.py references object_detection.configs or semantic_segmentation.configs
which register models and pipelines in MMDetection/MMSeg frameworks
- run_with_submitit wraps the entire flow for distributed scheduling

##### Architectural Patterns & Design Principles
- Modular tasks separation: classification vs. detection vs. segmentation
- Config‐driven workflows: Python config files under each task’s configs/ folder
- Plugin/extension of external frameworks (MMDetection, MMSegmentation) via mmcv_custom
- Separation of concerns: dataset, model, engine, optimizer, utils each in its own module
- Reuse of ConvNeXt core across all pipelines

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Entry and Execution
    EntryPoint["Entry Point\n(main.py)"]:::custom
    ClusterRunner["Cluster Runner\n(run_with_submitit.py)"]:::env

    %% Configuration
    subgraph "Configurations" 
        direction TB
        ConfigLoader["Config Loader\n(Python config files)"]:::data
        subgraph "Classification & Downstream Task Configs"
            direction TB
            DetConfigsBase["Detection Base Configs"]:::data
            DetConfigsConv["Detection ConvNeXt Configs"]:::data
            SegConfigsBase["Segmentation Base Configs"]:::data
            SegConfigsConv["Segmentation ConvNeXt Configs"]:::data
        end
    end

    %% Data Layer
    DatasetBuilder["Dataset Builder\n(datasets.py)"]:::custom

    %% Model Layer
    subgraph "Model Layer"
        direction TB
        ConvNeXtCore["ConvNeXt Core\n(models/convnext.py)"]:::custom
        ConvNeXtIso["ConvNeXt Isotropic\n(models/convnext_isotropic.py)"]:::custom
        subgraph "Detection Model Extensions"
            direction TB
            DetBackbone["Det Backbone\n(object_detection/mmdet/models/backbones/convnext.py)"]:::custom
            DetMMCV["Det mmcv_custom\n(customized_text.py)"]:::custom
            DetMMCV2["Det mmcv_custom\n(layer_decay_optimizer_constructor.py)"]:::custom
            DetMMCV3["Det mmcv_custom runner\n(checkpoint.py)"]:::custom
        end
        subgraph "Segmentation Model Extensions"
            direction TB
            SegBackbone["Seg Backbone\n(semantic_segmentation/backbone/convnext.py)"]:::custom
            SegMMCV1["Seg mmcv_custom runner\n(apex_runner/checkpoint.py)"]:::custom
            SegMMCV2["Seg mmcv_custom\n(customized_text.py)"]:::custom
            SegMMCV3["Seg mmcv_custom\n(layer_decay_optimizer_constructor.py)"]:::custom
        end
    end

    %% Engine and Optimization
    TrainingEngine["Training Engine\n(engine.py)"]:::custom
    OptimizerBuilder["Optimizer & Scheduler Factory\n(optim_factory.py)"]:::custom

    %% Utilities
    Utilities["Utilities & Helpers\n(utils.py)"]:::custom

    %% External Frameworks
    subgraph "External Frameworks"
        direction TB
        PyTorch["PyTorch"]:::external
        Timm["timm"]:::external
        MMDetection["MMDetection"]:::external
        MMSegmentation["MMSegmentation"]:::external
    end

    %% Relationships
    EntryPoint -->|"loads config"| ConfigLoader
    EntryPoint -->|"dispatches pipeline"| DatasetBuilder
    EntryPoint -->|"dispatches pipeline"| ConvNeXtCore
    EntryPoint -->|"dispatches pipeline"| OptimizerBuilder
    EntryPoint -->|"dispatches pipeline"| TrainingEngine
    EntryPoint -->|"distributed launch"| ClusterRunner
    ClusterRunner -->|"sets up env & resume"| EntryPoint

    ConfigLoader -->|"provides settings"| DatasetBuilder
    ConfigLoader -->|"provides settings"| ConvNeXtCore
    ConfigLoader -->|"provides settings"| OptimizerBuilder

    DatasetBuilder -->|"training data"| TrainingEngine
    ConvNeXtCore -->|"model weights"| TrainingEngine
    ConvNeXtIso -->|"model weights"| TrainingEngine
    DetBackbone -->|"model weights"| TrainingEngine
    SegBackbone -->|"model weights"| TrainingEngine

    OptimizerBuilder -->|"optimizer & schedulers"| TrainingEngine

    TrainingEngine -->|"logs & checkpoints"| Utilities
    Utilities -->|"checkpoint I/O"| ClusterRunner

    %% Plugins to frameworks
    ConvNeXtCore -->|"calls"| PyTorch
    Timm -->|"optional weights"| ConvNeXtCore
    DetBackbone -->|"integrates with"| MMDetection
    DetMMCV -->|"hooks into"| MMDetection
    MMSegmentation -->|"used by"| SegBackbone
    SegMMCV1 -->|"hooks into"| MMSegmentation

    %% Click Events
    click EntryPoint "https://github.com/facebookresearch/convnext/blob/main/main.py"
    click ClusterRunner "https://github.com/facebookresearch/convnext/blob/main/run_with_submitit.py"
    click TrainingEngine "https://github.com/facebookresearch/convnext/blob/main/engine.py"
    click DatasetBuilder "https://github.com/facebookresearch/convnext/blob/main/datasets.py"
    click OptimizerBuilder "https://github.com/facebookresearch/convnext/blob/main/optim_factory.py"
    click Utilities "https://github.com/facebookresearch/convnext/blob/main/utils.py"
    click ConvNeXtCore "https://github.com/facebookresearch/convnext/blob/main/models/convnext.py"
    click ConvNeXtIso "https://github.com/facebookresearch/convnext/blob/main/models/convnext_isotropic.py"
    click DetConfigsBase "https://github.com/facebookresearch/convnext/tree/main/object_detection/configs/_base_/"
    click DetConfigsConv "https://github.com/facebookresearch/convnext/tree/main/object_detection/configs/convnext/"
    click DetBackbone "https://github.com/facebookresearch/convnext/blob/main/object_detection/mmdet/models/backbones/convnext.py"
    click DetMMCV "https://github.com/facebookresearch/convnext/blob/main/object_detection/mmcv_custom/customized_text.py"
    click DetMMCV2 "https://github.com/facebookresearch/convnext/blob/main/object_detection/mmcv_custom/layer_decay_optimizer_constructor.py"
    click DetMMCV3 "https://github.com/facebookresearch/convnext/blob/main/object_detection/mmcv_custom/runner/checkpoint.py"
    click SegConfigsBase "https://github.com/facebookresearch/convnext/tree/main/semantic_segmentation/configs/_base_/"
    click SegConfigsConv "https://github.com/facebookresearch/convnext/tree/main/semantic_segmentation/configs/convnext/"
    click SegBackbone "https://github.com/facebookresearch/convnext/blob/main/semantic_segmentation/backbone/convnext.py"
    click SegMMCV1 "https://github.com/facebookresearch/convnext/blob/main/semantic_segmentation/mmcv_custom/apex_runner/checkpoint.py"
    click SegMMCV2 "https://github.com/facebookresearch/convnext/blob/main/semantic_segmentation/mmcv_custom/customized_text.py"
    click SegMMCV3 "https://github.com/facebookresearch/convnext/blob/main/semantic_segmentation/mmcv_custom/layer_decay_optimizer_constructor.py"

    %% Styles
    classDef custom fill:#AED6F1,stroke:#2980B9,color:#1B2631
    classDef external fill:#ABEBBC,stroke:#239B56,color:#1B2631
    classDef data fill:#F9E79F,stroke:#B7950B,color:#1B2631
    classDef env fill:#D5D8DC,stroke:#85929E,color:#1B2631
```
