How phamquiluan/residualmaskingnetwork works
1. Project purpose and primary runtime
- This is a PyTorch-based facial expression recognition / emotion detection repository centered on the Residual Masking Network (RMN) family of models.
- It supports both training and inference/demo workflows, with the public user-facing API exposed through the rmn package.
2. Main entrypoints
- main_fer2013.py is the current top-level training entrypoint for the FER2013 workflow.
- The legacy/ directory contains older experiment drivers and demos for other datasets and model variants; these are historical, not the main architectural path.
- The README also references a multi-GPU ImageNet entrypoint, but in the current tree the active, clearly documented entrypoint is the FER2013 one.
3. Core inference boundary
- The package rmn is the installable inference-facing boundary.
- The README shows the intended usage pattern: instantiate RMN, call detect_emotion_for_single_frame(...), then optionally draw(...) or video_demo().
- This suggests a clean separation between public API / demo runtime and the internal model zoo/training code.
4. Model zoo
- The models/ directory contains the main neural network implementations: ResNet, VGG, AlexNet, DenseNet, GoogLeNet, Inception, Inception-ResNet, and RMN-specific variants such as resmasking, resatt, attention, masking, and residual_attention_network.
- There is also a models/segmentation/ subtree with FCN, U-Net, and DeepLabV3 implementations, indicating the repository includes some auxiliary segmentation-capable architectures or reusable backbone components.
- The models/ and rmn/models/ trees are largely mirrored, which strongly suggests a source tree vs packaged distribution tree split.
5. Packaging/layout boundary
- The root models/ and rmn/models/ duplication indicates the project is structured both as a repository and as a publishable Python package.
- setup.py, pyproject.toml, and MANIFEST.in define packaging and installation.
- rmn/ is the importable package namespace exposed to users, while the root-level code appears to be the development/source workspace.
6. Training orchestration
- The trainers/ package contains the training control layer: a generic trainer.py, plus dataset/task-specific trainers such as _fer2013_trainer.py, ck_trainer.py, z_trainer.py, centerloss_trainer.py, dist_trainer.py, and tta_trainer.py.
- Architecturally, this is the layer that coordinates dataloading → augmentation → forward pass → loss/metrics → checkpointing.
- It also reflects multiple training modes: standard supervised training, center-loss training, distributed training, and test-time augmentation.
7. Dataset adapters
- utils/datasets/ contains dataset-specific loaders for affectnet, ck, fer2013, mixed_emotion, mnist, table, and z.
- These modules form the ingestion boundary between raw dataset formats and the trainer/model pipeline.
- The README specifically highlights FER2013, while the tree shows support for several emotion-recognition benchmarks and some utility datasets.
8. Preprocessing, augmentation, and supporting utilities
- utils/augmenters/ contains image augmentation logic, which likely feeds the training pipeline.
- utils/processing/, utils/generals.py, and utils/utils.py provide common helpers and data manipulation.
- utils/losses.py, utils/center_loss.py, and utils/radam.py provide reusable training components, including specialized losses and the optimizer implementation.
9. Evaluation and metrics
- utils/metrics/ contains standard classification metrics and segmentation metrics.
- The repository also includes scripts for result generation and ensemble evaluation in script/gen_results.py and script/gen_ensemble.py.
- This indicates an explicit offline evaluation flow, especially for comparing multiple trained checkpoints and combining predictions.
10. Configuration-driven experiments
- Experiment settings are stored in JSON configs under configs/, with a packaged copy under rmn/configs/.
- The README makes clear that model choice and hyperparameters are driven by these config files, especially for FER2013.
- This creates a boundary between code and experiment specification, which is important for reproducible training runs.
11. Legacy and research artifacts
- legacy/ contains older standalone scripts for demos, Grad-CAM visualizations, and previous training entrypoints on multiple datasets.
- These files show the project’s research history, but they are not the main production path; they are best treated as experimental/archival modules in an architecture diagram.
12. Operational/tooling layer
- CI and quality controls are present via .circleci/, GitHub Actions workflows, pre-commit, flake8, isort, and Black-style formatting conventions.
- This affects the repository only as a development/verification boundary, not the runtime architecture.
- The project is also designed for distribution via PyPI and includes documentation artifacts in docs/ and README.md.
13. Overall data flow
- The practical runtime path is: dataset loader → augmentation/processing → trainer → model forward pass → loss/metrics → checkpoint/output.
- For inference, the path is: OpenCV image/frame → RMN API → face/emotion prediction → annotated image or video demo output.
- The main architectural distinction is therefore between the training subsystem and the packaged inference API, both backed by the shared model zoo.
Main entry points: _fer2013_trainer.py FER trainer · resmasking.py ResMasking · __init__.py RMN API · metrics.py Metrics
Components 20
Training runtime
FER2013 training and evaluation orchestration
- FER2013 main · Top-level training entrypoint for the FER2013 workflow
main_fer2013.py - Trainer core · Shared training loop and control flow
trainers/trainer.py - FER trainer · FER2013-specific training logic
trainers/_fer2013_trainer.py - Center loss · Training mode with center-loss supervision
trainers/centerloss_trainer.py
Model zoo
Core network implementations used by training and inference
- ResMasking · RMN-family residual masking network
models/resmasking.py - ResAtt · Residual attention variant used in the model zoo
models/resatt.py - ResNet · General-purpose ResNet family implementations
models/resnet.py - Segmentation · Segmentation architectures and shared wrappers
models/segmentation/segmentation.py
Packaged API
Installable inference-facing namespace and mirrored package tree
- RMN API · Installable inference-facing boundary
rmn/__init__.py - Pkg ResMasking · Packaged mirror of the RMN core model
rmn/models/resmasking.py - Pkg ResAtt · Packaged mirror of the attention variant
rmn/models/resatt.py
Data and losses
Dataset adapters, preprocessing, metrics, and training primitives
- FER2013 data · FER2013 dataset loader boundary
utils/datasets/fer2013dataset.py - AffectNet data · Alternate emotion dataset loader
utils/datasets/affectnet_dataset.py - Augmenters · Image augmentation and transform logic
utils/augmenters/augment.py - Processing · Shared data manipulation and preprocessing helpers
utils/processing/__init__.py - Losses · Reusable loss functions and optimization helpers
utils/losses.py - Metrics · Classification and segmentation metric implementations
utils/metrics/metrics.py
Ops and scripts
Offline result generation and ensemble utilities
Experiment spec
Configuration files that select models and hyperparameters
- FER config · FER2013 experiment specification
rmn/configs/fer2013_config.json
Connections 21
- FER config → FER2013 main: drives · Model and hyperparameter selection for the training run
- FER2013 main → FER trainer: starts · Launches the FER2013 training workflow
- FER trainer → Trainer core: extends · Task-specific trainer builds on the shared loop
- FER trainer → FER2013 data: loads · Consumes FER2013 samples and labels
- FER trainer → Augmenters: augments · Applies image augmentation before training
- FER trainer → Processing: preprocesses · Uses shared preprocessing helpers
- FER trainer → ResMasking: trains · Primary RMN-family training target
- Center loss → Losses: uses · Adds specialized center-loss objective
- FER trainer → Losses: optimizes · Computes supervised training losses
- FER trainer → Metrics: measures · Tracks model quality during training and validation
- ResMasking → ResAtt: relates · Adjacent RMN-family architecture variant
- ResMasking → ResNet: builds on · Shares residual backbone ideas with the broader model zoo
- ResMasking → Segmentation: shares · Repository also ships auxiliary segmentation architectures
- RMN API → Pkg ResMasking: exposes · Public package surface for inference
- RMN API → Pkg ResAtt: exposes · Public package surface for inference
- Pkg ResMasking → ResMasking: mirrors · Packaged copy of the core model implementation
- Pkg ResAtt → ResAtt: mirrors · Packaged copy of the attention variant
- RMN API → FER config: configured by · Inference and demo behavior are also config-driven
- Results → Metrics: summarizes · Offline evaluation and report generation
- Ensemble → Metrics: combines · Ensemble scoring and comparison workflow
- AffectNet data → Trainer core: feeds · Alternative dataset source for trainer variants