How rt219/the-emergence-of-objectness works
Project Type and Purpose
• This repository is a research‐oriented deep learning framework built on top of MMSegmentation (a PyTorch-based semantic segmentation toolbox).
• Purpose: Implement and train an unsupervised “zero‐shot” object segmentation model from videos, combining an appearance pathway (single‐image segmentation) and a motion pathway (dual‐frame flow estimation).
High-Level File Structure
• configs/
Contains Python config files (config_train.py, config_test.py) that specify dataset paths, model hyperparameters, optimization schedules, and runtime settings (e.g. distributed training ports).
• data_example/
Example dataset layout (Annotations, JPEGImages, train/inference split files) used for quick testing and demo.
• mmseg/
Core library, subdivided into:
– apis/ (entry points: train.py, test.py, inference.py)
– core/ (training loop hooks, evaluation metrics, builder patterns, distributed utils)
– datasets/ (dataset factories, pipeline transforms, custom wrappers)
– models/ (backbones, decode heads, segmentors, loss functions, utility modules)
– ops/ (custom CUDA ops, separable‐conv modules)
– utils/ (logging, environment collection, general helpers)
• tools/
Utility scripts: benchmark, dataset converters, ONNX export, multi‐GPU launch wrappers (dist_train.sh / dist_test.sh), Slurm helpers.
• tests/
Unit tests verifying config parsing, data pipelines, model forward logic, ops correctness, evaluation computations.
Key Architectural Patterns
• Registry/Builder Pattern: All datasets, models, backbones, heads, losses are registered via factory builders so new components plug in seamlessly.
• Modular Pipelines: Data loading → transforms → batch sampling → pixel sampler → model ingestion.
• Separation of Concerns:
• Configs are externalized.
• Core library is agnostic to any specific dataset or model.
• Scripts (in apis/ and tools/) compose the workflow.
Main Components and Their Interactions
A. Config Manager (configs/)
• Parses and validates training or testing configuration.
• Provides a unified config object to all downstream components.
B. Data Ingestion
• Dataset Builder (mmseg/datasets/builder.py) reads data_root and split from config.
• Pipeline of transforms (loading, resizing, normalization) defined in mmseg/datasets/pipelines.
• Pixel Sampler (OHEM or random) selects pixels for loss computation.
C. Model Assembly
• Backbone (ResNet, etc.) → Neck (FPN) → Decode Heads → Loss functions (cross‐entropy, flow loss)
• Segmentor class glues appearance and motion pathways into “segment flow” representation.
D. Training and Evaluation Engine (mmseg/apis/train.py, test.py)
• Instantiates data loaders (incl. distributed sampler), model, optimizer, scheduler.
• Hooks: checkpointing, logging, evaluation (mean IoU) per interval.
• Loss backprop and weight updates.
E. Inference & Deployment (mmseg/apis/inference.py, tools/pytorch2onnx.py)
• Loads pretrained weights, runs forward pass on single images or frame pairs.
• Exports to ONNX if needed, writes outputs to disk for visualization.
F. External Services & Tools
• Slurm / multi‐GPU launch: tools/dist_train.sh, slurm_train.sh
• Dataset conversion utilities for Cityscapes, VOC (tools/convert_datasets/)
• Benchmarking and FLOPs computation scripts.