# rt219/the-emergence-of-objectness architecture

> Architecture diagram of the GitHub repository rt219/the-emergence-of-objectness, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rt219/the-emergence-of-objectness
- Repository: https://github.com/rt219/the-emergence-of-objectness

## Overview

##### Project Type and Purpose
- This repository is a research‐oriented deep learning framework built on top of MMSegmentation (a PyTorch-based semantic segmentation toolbox).
- Purpose: Implement and train an unsupervised “zero‐shot” object segmentation model from videos, combining an appearance pathway (single‐image segmentation) and a motion pathway (dual‐frame flow estimation).

##### High-Level File Structure
- configs/
Contains Python config files (`config_train.py`, `config_test.py`) that specify dataset paths, model hyperparameters, optimization schedules, and runtime settings (e.g. distributed training ports).
- data_example/
Example dataset layout (Annotations, JPEGImages, train/inference split files) used for quick testing and demo.
- mmseg/
Core library, subdivided into:
– apis/ (entry points: `train.py`, `test.py`, `inference.py`)
– core/ (training loop hooks, evaluation metrics, builder patterns, distributed utils)
– datasets/ (dataset factories, pipeline transforms, custom wrappers)
– models/ (backbones, decode heads, segmentors, loss functions, utility modules)
– ops/ (custom CUDA ops, separable‐conv modules)
– utils/ (logging, environment collection, general helpers)
- tools/
Utility scripts: benchmark, dataset converters, ONNX export, multi‐GPU launch wrappers (`dist_train.sh` / `dist_test.sh`), Slurm helpers.
- tests/
Unit tests verifying config parsing, data pipelines, model forward logic, ops correctness, evaluation computations.

##### Key Architectural Patterns
- **Registry/Builder Pattern**: All datasets, models, backbones, heads, losses are registered via factory builders so new components plug in seamlessly.
- **Modular Pipelines**: Data loading → transforms → batch sampling → pixel sampler → model ingestion.
- **Separation of Concerns**:
- Configs are externalized.
- Core library is agnostic to any specific dataset or model.
- Scripts (in apis/ and tools/) compose the workflow.

##### Main Components and Their Interactions
A. Config Manager (configs/)
- Parses and validates training or testing configuration.
- Provides a unified config object to all downstream components.
B. Data Ingestion
- Dataset Builder (mmseg/datasets/builder.py) reads `data_root` and `split` from config.
- Pipeline of transforms (loading, resizing, normalization) defined in mmseg/datasets/pipelines.
- Pixel Sampler (OHEM or random) selects pixels for loss computation.
C. Model Assembly
- Backbone (ResNet, etc.) → Neck (FPN) → Decode Heads → Loss functions (cross‐entropy, flow loss)
- Segmentor class glues appearance and motion pathways into “segment flow” representation.
D. Training and Evaluation Engine (mmseg/apis/train.py, test.py)
- Instantiates data loaders (incl. distributed sampler), model, optimizer, scheduler.
- Hooks: checkpointing, logging, evaluation (mean IoU) per interval.
- Loss backprop and weight updates.
E. Inference & Deployment (mmseg/apis/inference.py, tools/pytorch2onnx.py)
- Loads pretrained weights, runs forward pass on single images or frame pairs.
- Exports to ONNX if needed, writes outputs to disk for visualization.
F. External Services & Tools
- Slurm / multi‐GPU launch: `tools/dist_train.sh`, `slurm_train.sh`
- Dataset conversion utilities for Cityscapes, VOC (`tools/convert_datasets/`)
- Benchmarking and FLOPs computation scripts.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Components
    ConfigManager["Config Manager"]:::data
    click ConfigManager "https://github.com/rt219/the-emergence-of-objectness/blob/main/configs/config_train.py"
    click ConfigManager "https://github.com/rt219/the-emergence-of-objectness/blob/main/configs/config_test.py"

    subgraph "Data Ingestion" 
        direction TB
        DatasetBuilder["Dataset Builder"]:::data
        click DatasetBuilder "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/datasets/builder.py"
        subgraph "Pipelines"
            direction TB
            Compose["Compose"]:::data
            click Compose "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/datasets/pipelines/compose.py"
            Loading["Loading"]:::data
            click Loading "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/datasets/pipelines/loading.py"
            Transforms["Transforms"]:::data
            click Transforms "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/datasets/pipelines/transforms.py"
        end
        subgraph "Pixel Sampling"
            direction TB
            BaseSampler["BaseSampler"]:::data
            click BaseSampler "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/core/seg/sampler/base_pixel_sampler.py"
            OHEMSampler["OHEMSampler"]:::data
            click OHEMSampler "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/core/seg/sampler/ohem_pixel_sampler.py"
        end
    end

    %% Model Components
    subgraph "Model Assembly" 
        direction TB
        ModelBuilder["Model Builder & Registry"]:::compute
        click ModelBuilder "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/models/builder.py"
        subgraph "Backbone & Neck"
            direction TB
            Backbone["ResNet Backbone"]:::compute
            click Backbone "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/models/backbones/resnet.py"
            Neck["FPN Neck"]:::compute
            click Neck "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/models/necks/fpn.py"
        end
        subgraph "Decode Heads & Loss"
            direction TB
            DecodeHead["Decode Head"]:::compute
            click DecodeHead "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/models/decode_heads/decode_head.py"
            FCNHead["FCN Head"]:::compute
            click FCNHead "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/models/decode_heads/fcn_head.py"
            FlowLoss["Flow Loss"]:::compute
            click FlowLoss "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/models/decode_heads/losses/flow_loss.py"
        end
        Segmentor["Segmentor (A&M Fusion)"]:::compute
        click Segmentor "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/models/segmentors/encoder_decoder.py"
    end

    %% Training & Evaluation
    subgraph "Training & Evaluation" 
        direction TB
        Trainer["Training API & Loop"]:::compute
        click Trainer "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/apis/train.py"
        EvalHooks["Eval Hooks"]:::compute
        click EvalHooks "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/core/evaluation/eval_hooks.py"
        DistUtils["Distributed Utils"]:::compute
        click DistUtils "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/core/utils/dist_utils.py"
        Tester["Evaluation API & Metrics"]:::compute
        click Tester "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/apis/test.py"
        MeanIoU["Mean IoU Metric"]:::compute
        click MeanIoU "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/core/evaluation/mean_iou.py"
    end

    %% Inference & Tools
    subgraph "Inference & Tools"
        direction TB
        Inference["Inference API"]:::deploy
        click Inference "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/apis/inference.py"
        ONNXExport["ONNX Export"]:::deploy
        click ONNXExport "https://github.com/rt219/the-emergence-of-objectness/blob/main/tools/pytorch2onnx.py"
        subgraph "Scripts"
            direction TB
            DistTrain["dist_train.sh"]:::deploy
            click DistTrain "https://github.com/rt219/the-emergence-of-objectness/blob/main/tools/dist_train.sh"
            SlurmTrain["slurm_train.sh"]:::deploy
            click SlurmTrain "https://github.com/rt219/the-emergence-of-objectness/blob/main/tools/slurm_train.sh"
            DistTest["dist_test.sh"]:::deploy
            click DistTest "https://github.com/rt219/the-emergence-of-objectness/blob/main/tools/dist_test.sh"
            Cityscapes["cityscapes.py"]:::deploy
            click Cityscapes "https://github.com/rt219/the-emergence-of-objectness/blob/main/tools/convert_datasets/cityscapes.py"
            VOC["voc_aug.py"]:::deploy
            click VOC "https://github.com/rt219/the-emergence-of-objectness/blob/main/tools/convert_datasets/voc_aug.py"
            Benchmark["benchmark.py"]:::deploy
            click Benchmark "https://github.com/rt219/the-emergence-of-objectness/blob/main/tools/benchmark.py"
            Flops["get_flops.py"]:::deploy
            click Flops "https://github.com/rt219/the-emergence-of-objectness/blob/main/tools/get_flops.py"
            PrintConfig["print_config.py"]:::deploy
            click PrintConfig "https://github.com/rt219/the-emergence-of-objectness/blob/main/tools/print_config.py"
            Publish["publish_model.py"]:::deploy
            click Publish "https://github.com/rt219/the-emergence-of-objectness/blob/main/tools/publish_model.py"
            TrainScript["train.py"]:::deploy
            click TrainScript "https://github.com/rt219/the-emergence-of-objectness/blob/main/tools/train.py"
        end
        EnvLogger["Env & Logger Utils"]:::compute
        click EnvLogger "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/utils/collect_env.py"
        click EnvLogger "https://github.com/rt219/the-emergence-of-objectness/blob/main/mmseg/utils/logger.py"
    end

    %% External Dependencies
    PyTorch["PyTorch"]:::external
    CUDA["CUDA"]:::external
    ONNX["ONNX"]:::external
    Slurm["Slurm"]:::external

    %% Data Flow
    ConfigManager --> DatasetBuilder --> Compose --> Loading --> Transforms --> DatasetBuilder
    DatasetBuilder --> BaseSampler --> OHEMSampler
    DatasetBuilder --> Trainer
    ModelBuilder --> Backbone --> Neck --> DecodeHead --> FCNHead
    FCNHead --> FlowLoss --> Segmentor
    Segmentor --> Trainer
    Trainer --> EvalHooks --> Tester --> MeanIoU
    Trainer --> DistUtils
    Trainer --> ModelBuilder
    Trainer --> Optimizer[Optimizer]:::compute
    Optimizer --> Segmentor
    Trainer --> CKpt[Checkpoint Hook]:::compute
    Trainer --> Logger[Logger Hook]:::compute

    Inference --> Segmentor --> ONNXExport --> ONNX
    Inference --> EnvLogger
    DistTrain --> Trainer
    SlurmTrain --> Trainer
    DistTest --> Tester

    %% External Services
    Trainer --- PyTorch
    Segmentor --- CUDA
    ONNXExport --- ONNX
    DistTrain --- Slurm
    SlurmTrain --- Slurm

    classDef data fill:#d5f5e3,stroke:#27ae60,color:#000
    classDef compute fill:#d6eaf8,stroke:#2980b9,color:#000
    classDef deploy fill:#f9e79f,stroke:#d68910,color:#000
    classDef external fill:#f2f4f4,stroke:#95a5a6,color:#000
```
