# phamquiluan/residualmaskingnetwork architecture

> Architecture diagram of the GitHub repository phamquiluan/residualmaskingnetwork, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/phamquiluan/residualmaskingnetwork
- Repository: https://github.com/phamquiluan/residualmaskingnetwork

## Overview

1. **Project purpose and primary runtime**
   - This is a **PyTorch-based facial expression recognition / emotion detection** repository centered on the **Residual Masking Network (RMN)** family of models.
   - It supports both **training** and **inference/demo** workflows, with the public user-facing API exposed through the `rmn` package.

2. **Main entrypoints**
   - `main_fer2013.py` is the current top-level training entrypoint for the FER2013 workflow.
   - The `legacy/` directory contains older experiment drivers and demos for other datasets and model variants; these are historical, not the main architectural path.
   - The README also references a multi-GPU ImageNet entrypoint, but in the current tree the active, clearly documented entrypoint is the FER2013 one.

3. **Core inference boundary**
   - The package `rmn` is the installable inference-facing boundary.
   - The README shows the intended usage pattern: instantiate `RMN`, call `detect_emotion_for_single_frame(...)`, then optionally `draw(...)` or `video_demo()`.
   - This suggests a clean separation between **public API / demo runtime** and the internal model zoo/training code.

4. **Model zoo**
   - The `models/` directory contains the main neural network implementations: ResNet, VGG, AlexNet, DenseNet, GoogLeNet, Inception, Inception-ResNet, and RMN-specific variants such as `resmasking`, `resatt`, `attention`, `masking`, and `residual_attention_network`.
   - There is also a `models/segmentation/` subtree with FCN, U-Net, and DeepLabV3 implementations, indicating the repository includes some auxiliary segmentation-capable architectures or reusable backbone components.
   - The `models/` and `rmn/models/` trees are largely mirrored, which strongly suggests a **source tree vs packaged distribution tree** split.

5. **Packaging/layout boundary**
   - The root `models/` and `rmn/models/` duplication indicates the project is structured both as a repository and as a publishable Python package.
   - `setup.py`, `pyproject.toml`, and `MANIFEST.in` define packaging and installation.
   - `rmn/` is the importable package namespace exposed to users, while the root-level code appears to be the development/source workspace.

6. **Training orchestration**
   - The `trainers/` package contains the training control layer: a generic `trainer.py`, plus dataset/task-specific trainers such as `_fer2013_trainer.py`, `ck_trainer.py`, `z_trainer.py`, `centerloss_trainer.py`, `dist_trainer.py`, and `tta_trainer.py`.
   - Architecturally, this is the layer that coordinates **dataloading → augmentation → forward pass → loss/metrics → checkpointing**.
   - It also reflects multiple training modes: standard supervised training, center-loss training, distributed training, and test-time augmentation.

7. **Dataset adapters**
   - `utils/datasets/` contains dataset-specific loaders for `affectnet`, `ck`, `fer2013`, `mixed_emotion`, `mnist`, `table`, and `z`.
   - These modules form the ingestion boundary between raw dataset formats and the trainer/model pipeline.
   - The README specifically highlights FER2013, while the tree shows support for several emotion-recognition benchmarks and some utility datasets.

8. **Preprocessing, augmentation, and supporting utilities**
   - `utils/augmenters/` contains image augmentation logic, which likely feeds the training pipeline.
   - `utils/processing/`, `utils/generals.py`, and `utils/utils.py` provide common helpers and data manipulation.
   - `utils/losses.py`, `utils/center_loss.py`, and `utils/radam.py` provide reusable training components, including specialized losses and the optimizer implementation.

9. **Evaluation and metrics**
   - `utils/metrics/` contains standard classification metrics and segmentation metrics.
   - The repository also includes scripts for result generation and ensemble evaluation in `script/gen_results.py` and `script/gen_ensemble.py`.
   - This indicates an explicit offline evaluation flow, especially for comparing multiple trained checkpoints and combining predictions.

10. **Configuration-driven experiments**
   - Experiment settings are stored in JSON configs under `configs/`, with a packaged copy under `rmn/configs/`.
   - The README makes clear that model choice and hyperparameters are driven by these config files, especially for FER2013.
   - This creates a boundary between **code** and **experiment specification**, which is important for reproducible training runs.

11. **Legacy and research artifacts**
   - `legacy/` contains older standalone scripts for demos, Grad-CAM visualizations, and previous training entrypoints on multiple datasets.
   - These files show the project’s research history, but they are not the main production path; they are best treated as experimental/archival modules in an architecture diagram.

12. **Operational/tooling layer**
   - CI and quality controls are present via `.circleci/`, GitHub Actions workflows, `pre-commit`, `flake8`, `isort`, and Black-style formatting conventions.
   - This affects the repository only as a development/verification boundary, not the runtime architecture.
   - The project is also designed for distribution via PyPI and includes documentation artifacts in `docs/` and `README.md`.

13. **Overall data flow**
   - The practical runtime path is: **dataset loader → augmentation/processing → trainer → model forward pass → loss/metrics → checkpoint/output**.
   - For inference, the path is: **OpenCV image/frame → RMN API → face/emotion prediction → annotated image or video demo output**.
   - The main architectural distinction is therefore between the **training subsystem** and the **packaged inference API**, both backed by the shared model zoo.

Main entry points: [`trainers/_fer2013_trainer.py`](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/trainers/_fer2013_trainer.py) (FER trainer), [`models/resmasking.py`](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/models/resmasking.py) (ResMasking), [`rmn/__init__.py`](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/rmn/__init__.py) (RMN API), [`utils/metrics/metrics.py`](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/utils/metrics/metrics.py) (Metrics)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Training runtime"]
  node_main_fer2013["FER2013 main<br/>entrypoint<br/>[main_fer2013.py]"]
  node_trainer_base["Trainer core<br/>training orchestrator<br/>[trainer.py]"]
  node_trainer_fer2013["FER trainer<br/>task trainer"]
  node_trainer_center["Center loss<br/>training variant"]
end

subgraph group_models["Model zoo"]
  node_resmasking["ResMasking<br/>emotion model<br/>[resmasking.py]"]
  node_resatt["ResAtt<br/>attention model<br/>[resatt.py]"]
  node_resnet["ResNet<br/>backbone<br/>[resnet.py]"]
  node_segmentation["Segmentation<br/>aux models<br/>[segmentation.py]"]
end

subgraph group_package["Packaged API"]
  node_rmn_api["RMN API<br/>public package<br/>[__init__.py]"]
  node_pkg_resmasking["Pkg ResMasking<br/>packaged model<br/>[resmasking.py]"]
  node_pkg_resatt["Pkg ResAtt<br/>packaged model<br/>[resatt.py]"]
end

subgraph group_data["Data and losses"]
  node_fer_dataset["FER2013 data<br/>dataset adapter<br/>[fer2013dataset.py]"]
  node_affectnet_dataset["AffectNet data<br/>dataset adapter"]
  node_augmenters["Augmenters<br/>image augmentation<br/>[augment.py]"]
  node_processing["Processing<br/>[__init__.py]"]
  node_losses["Losses<br/>training primitives<br/>[losses.py]"]
  node_metrics["Metrics<br/>evaluation<br/>[metrics.py]"]
end

subgraph group_ops["Ops and scripts"]
  node_gen_results["Results<br/>offline eval<br/>[gen_results.py]"]
  node_gen_ensemble["Ensemble<br/>offline eval<br/>[gen_ensemble.py]"]
end

subgraph group_spec["Experiment spec"]
  node_config_fer["FER config<br/>experiment config"]
end

node_config_fer -->|"drives"| node_main_fer2013
node_main_fer2013 -->|"starts"| node_trainer_fer2013
node_trainer_fer2013 -->|"extends"| node_trainer_base
node_trainer_fer2013 -->|"loads"| node_fer_dataset
node_trainer_fer2013 -->|"augments"| node_augmenters
node_trainer_fer2013 -->|"preprocesses"| node_processing
node_trainer_fer2013 -->|"trains"| node_resmasking
node_trainer_center -->|"uses"| node_losses
node_trainer_fer2013 -->|"optimizes"| node_losses
node_trainer_fer2013 -->|"measures"| node_metrics
node_resmasking -.->|"relates"| node_resatt
node_resmasking -.->|"builds on"| node_resnet
node_resmasking -.->|"shares"| node_segmentation
node_rmn_api -->|"exposes"| node_pkg_resmasking
node_rmn_api -->|"exposes"| node_pkg_resatt
node_pkg_resmasking -.->|"mirrors"| node_resmasking
node_pkg_resatt -.->|"mirrors"| node_resatt
node_rmn_api -.->|"configured by"| node_config_fer
node_gen_results -->|"summarizes"| node_metrics
node_gen_ensemble -->|"combines"| node_metrics
node_affectnet_dataset -.->|"feeds"| node_trainer_base

click node_main_fer2013 "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/main_fer2013.py"
click node_trainer_base "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/trainers/trainer.py"
click node_trainer_fer2013 "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/trainers/_fer2013_trainer.py"
click node_trainer_center "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/trainers/centerloss_trainer.py"
click node_fer_dataset "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/utils/datasets/fer2013dataset.py"
click node_affectnet_dataset "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/utils/datasets/affectnet_dataset.py"
click node_augmenters "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/utils/augmenters/augment.py"
click node_processing "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/utils/processing/__init__.py"
click node_losses "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/utils/losses.py"
click node_metrics "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/utils/metrics/metrics.py"
click node_resmasking "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/models/resmasking.py"
click node_resatt "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/models/resatt.py"
click node_resnet "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/models/resnet.py"
click node_segmentation "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/models/segmentation/segmentation.py"
click node_rmn_api "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/rmn/__init__.py"
click node_pkg_resmasking "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/rmn/models/resmasking.py"
click node_pkg_resatt "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/rmn/models/resatt.py"
click node_config_fer "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/rmn/configs/fer2013_config.json"
click node_gen_results "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/script/gen_results.py"
click node_gen_ensemble "https://github.com/phamquiluan/residualmaskingnetwork/blob/master/script/gen_ensemble.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_fer2013,node_trainer_base,node_trainer_fer2013,node_trainer_center toneBlue
class node_resmasking,node_resatt,node_resnet,node_segmentation toneAmber
class node_rmn_api,node_pkg_resmasking,node_pkg_resatt toneMint
class node_fer_dataset,node_affectnet_dataset,node_augmenters,node_processing,node_losses,node_metrics toneRose
class node_gen_results,node_gen_ensemble toneIndigo
class node_config_fer toneTeal
```

## Components

### Training runtime

FER2013 training and evaluation orchestration

- [FER2013 main](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/main_fer2013.py): entrypoint · Top-level training entrypoint for the FER2013 workflow · `main_fer2013.py`
- [Trainer core](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/trainers/trainer.py): training orchestrator · Shared training loop and control flow · `trainers/trainer.py`
- [FER trainer](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/trainers/_fer2013_trainer.py): task trainer · FER2013-specific training logic · `trainers/_fer2013_trainer.py`
- [Center loss](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/trainers/centerloss_trainer.py): training variant · Training mode with center-loss supervision · `trainers/centerloss_trainer.py`

### Model zoo

Core network implementations used by training and inference

- [ResMasking](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/models/resmasking.py): emotion model · RMN-family residual masking network · `models/resmasking.py`
- [ResAtt](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/models/resatt.py): attention model · Residual attention variant used in the model zoo · `models/resatt.py`
- [ResNet](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/models/resnet.py): backbone · General-purpose ResNet family implementations · `models/resnet.py`
- [Segmentation](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/models/segmentation/segmentation.py): aux models · Segmentation architectures and shared wrappers · `models/segmentation/segmentation.py`

### Packaged API

Installable inference-facing namespace and mirrored package tree

- [RMN API](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/rmn/__init__.py): public package · Installable inference-facing boundary · `rmn/__init__.py`
- [Pkg ResMasking](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/rmn/models/resmasking.py): packaged model · Packaged mirror of the RMN core model · `rmn/models/resmasking.py`
- [Pkg ResAtt](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/rmn/models/resatt.py): packaged model · Packaged mirror of the attention variant · `rmn/models/resatt.py`

### Data and losses

Dataset adapters, preprocessing, metrics, and training primitives

- [FER2013 data](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/utils/datasets/fer2013dataset.py): dataset adapter · FER2013 dataset loader boundary · `utils/datasets/fer2013dataset.py`
- [AffectNet data](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/utils/datasets/affectnet_dataset.py): dataset adapter · Alternate emotion dataset loader · `utils/datasets/affectnet_dataset.py`
- [Augmenters](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/utils/augmenters/augment.py): image augmentation · Image augmentation and transform logic · `utils/augmenters/augment.py`
- [Processing](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/utils/processing/__init__.py): preprocessing · Shared data manipulation and preprocessing helpers · `utils/processing/__init__.py`
- [Losses](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/utils/losses.py): training primitives · Reusable loss functions and optimization helpers · `utils/losses.py`
- [Metrics](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/utils/metrics/metrics.py): evaluation · Classification and segmentation metric implementations · `utils/metrics/metrics.py`

### Ops and scripts

Offline result generation and ensemble utilities

- [Results](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/script/gen_results.py): offline eval · Script for generating result summaries · `script/gen_results.py`
- [Ensemble](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/script/gen_ensemble.py): offline eval · Script for combining model predictions · `script/gen_ensemble.py`

### Experiment spec

Configuration files that select models and hyperparameters

- [FER config](https://github.com/phamquiluan/residualmaskingnetwork/blob/master/rmn/configs/fer2013_config.json): experiment config · FER2013 experiment specification · `rmn/configs/fer2013_config.json`

## Connections

- FER config → FER2013 main: drives · Model and hyperparameter selection for the training run
- FER2013 main → FER trainer: starts · Launches the FER2013 training workflow
- FER trainer → Trainer core: extends · Task-specific trainer builds on the shared loop
- FER trainer → FER2013 data: loads · Consumes FER2013 samples and labels
- FER trainer → Augmenters: augments · Applies image augmentation before training
- FER trainer → Processing: preprocesses · Uses shared preprocessing helpers
- FER trainer → ResMasking: trains · Primary RMN-family training target
- Center loss → Losses: uses · Adds specialized center-loss objective
- FER trainer → Losses: optimizes · Computes supervised training losses
- FER trainer → Metrics: measures · Tracks model quality during training and validation
- ResMasking → ResAtt: relates · Adjacent RMN-family architecture variant
- ResMasking → ResNet: builds on · Shares residual backbone ideas with the broader model zoo
- ResMasking → Segmentation: shares · Repository also ships auxiliary segmentation architectures
- RMN API → Pkg ResMasking: exposes · Public package surface for inference
- RMN API → Pkg ResAtt: exposes · Public package surface for inference
- Pkg ResMasking → ResMasking: mirrors · Packaged copy of the core model implementation
- Pkg ResAtt → ResAtt: mirrors · Packaged copy of the attention variant
- RMN API → FER config: configured by · Inference and demo behavior are also config-driven
- Results → Metrics: summarizes · Offline evaluation and report generation
- Ensemble → Metrics: combines · Ensemble scoring and comparison workflow
- AffectNet data → Trainer core: feeds · Alternative dataset source for trainer variants
