How nv-tlabs/gscnn works
Main Components
• Data Handling & Preprocessing:
• The datasets directory (with cityscapes.py, cityscapes_labels.py, edge_utils.py) indicates implementations for data loading, label processing, and possibly edge-related processing.
• The transforms directory (joint_transforms.py, transforms.py) deals with data augmentation and preprocessing.
• Network Architecture:
• The network directory includes various network modules such as Resnet.py, SEresnext.py, gscnn.py, mynn.py, and wider_resnet.py. This illustrates multiple backbone options and the main GSCNN implementation.
• The my_functionals directory contains customized modules (e.g., DualTaskLoss.py, GatedSpatialConv.py, custom_functional.py) that define specialized operations and loss functions critical for the gated mechanism.
• Training & Optimization:
• The train.py script is the entry point for training and evaluation.
• Files such as loss.py and optimizer.py encapsulate the loss calculations and optimization routines.
• Utilities & Configuration:
• config.py for configuration settings.
• The utils directory (with files like AttrDict.py, f_boundary.py, image_page.py, misc.py) provides helper functions and data handling utilities.
• Documentation & Resources:
• The docs directory includes documentation and additional media (like videos and gif demos) to help understand the system.
Relationships and Data Flow
• Input Data & Preprocessing:
• The flow begins with data loading from the datasets modules. Data is then passed to the transforms pipeline for augmentation.
• Model Pipeline:
• The preprocessed data is fed into the network’s architecture modules which combine backbone elements (ResNet variants, SEResNeXt, etc.) and the Gated-SCNN specific layers from my_functionals.
• Custom operations like gated spatial convolutions modify intermediate feature maps.
• Loss and Optimization:
• Outputs from the network are fed into loss functions (loss.py, DualTaskLoss.py, and others) that compute task-specific errors.
• The optimizer (optimizer.py) then uses these losses to update model weights.
• Training Controller:
• The train.py script orchestrates training and evaluation cycles across multiple GPUs (noted in the README) and interacts with model components and configuration parameters.
Architectural Patterns and Key Principles
• Modularity:
• The separation of data, network architecture, custom functionals, transforms, and utilities is a good example of modular design.
• Customization and Extensibility:
• By isolating custom operations in the my_functionals module and utilizing various network backbones, the design facilitates experimentation with architectural variants.
• Scalability:
• The project is designed to work on multiple GPUs for training (as indicated by the usage notes).
• Clear Separation of Concerns:
• Division between data processing (datasets/transforms), model definition (network/my_functionals), optimization (loss/optimizer), and execution control (train.py).