# hamadya/ghostfacenets architecture

> Architecture diagram of the GitHub repository hamadya/ghostfacenets, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hamadya/ghostfacenets
- Repository: https://github.com/hamadya/ghostfacenets

## Overview

##### Project Type and Purpose
- GhostFaceNets is a research‐oriented, full-Python deep‐learning codebase for lightweight face recognition models (GhostFaceNetV1/V2) built on TensorFlow/Keras. It provides end-to-end functionality: data preparation, model definition, training with flexible schedulers and callbacks, checkpointing, and evaluation on standard face-recognition benchmarks (LFW, IJB, MegaFace).

##### Main Components
A. Data & Preprocessing
- Datasets directory (`datasets/`): holds raw `.bin` files and extracted image folders for MS1MV2/MS1MV3, LFW, CFP-FP, AgeDB-30.
- prepare_data.py: converts MXNet `.bin` files to folder structure and reformats evaluation bins.
- augment.py: implements RandAugment/AutoAug transforms.
- data.py / data_distiller.py / data_drop_top_k.py: build `tf.data.Dataset` pipelines, support Triplet data, partial-FC splitting, sub-center ArcFace.

B. Model Definition
- backbones/ghost_model.py, ghostv2.py: core GhostNetV1/V2 architectures (copied & modified from keras_insightface).
- GhostFaceNets.py: wraps backbones, exposes buildin_models(), add_l2_regularizer, replace_ReLU_with_PReLU.
- models.py: utility to assemble backbone + embedding + output layer (GDC), and to print available models.

C. Training Pipeline
- train.py (`Train` class): orchestrates training. Takes data_path, eval_paths, basic_model/model, learning-rate schedule, batch size, callbacks, and implements:
– build model (basic_model → add bottleneck → compile)
– fit() with `tf.data.Dataset`, scheduler list of dicts (loss, optimizer, epochs)
- losses.py: ArcFace, CosFace, Subcenter ArcFace loss functions.
- myCallbacks.py: custom Keras callbacks (checkpoint saving, LR adjustment, history logging, “gently stop”).
- checkpoints/: stores `model.h5` per epoch and best basic_model.

D. Evaluation Modules
- evals.py: Keras callback to evaluate `.bin` files at end of epoch or batch.
- IJB_evals.py: command-line tool to evaluate on IJB-B/C datasets, generate embeddings, run protocols.
- eval_folder.py: evaluate on arbitrary folder format, also dump new `.bin` files.
- MegaFace/: scripts & notebooks (gen_megaface*, run.sh) to evaluate on MegaFace, handle noise removal, OpenCV2.4 dependency.

E. Notebooks & Utilities
- .ipynb files: Flops.ipynb, evaluation.ipynb, training_scripts.ipynb/training_scripts_with_Bias.ipynb – illustrate training/evaluation workflows interactively.
- plot_parameters.py: generate publication figures from saved history.
- video_test.py: demo real-time face recognition on video streams.

F. External Dependencies
- TensorFlow 2.8+, Keras 2.8+, keras_cv_attention_models, sklearn, pandas, glob2, scikit-image, tqdm, OpenCV2.4 (for MegaFace devkit), TensorFlow-Addons (optional optimizers).

##### Relationships & Data Flow
[ Data Sources ]
↓ prepare_data.py
[ Folder / Bin Inputs ]
↓ data.py + augment.py → tf.data.Dataset
↘
[ Train Pipeline ] ←─ GhostFaceNets.buildin_models + models.add_l2_regularizer
↙         ↓ compile & fit (scheduler, losses, optimizers)
[ Callbacks ] → checkpoint save to /checkpoints & history.json
↓
[ Saved Models ]
↓ evals.py / eval_folder.py / IJB_evals.py / MegaFace/run.sh
[ Evaluation Results & Plots ]

##### Architectural Patterns & Principles
- Modular separation: data ingestion, model definition, training engine, evaluation engine.
- Callback pattern: Keras callbacks for extensibility (evaluation hooks, checkpointing, LR scheduling).
- Scheduler pattern: list of training stages allowing dynamic switching of loss functions and optimizers.
- OOP for Train class: encapsulates state (datasets, model, schedulers, callbacks).
- Reusability: backbone definitions decoupled from output layers, added via utility APIs.

##### Final Checklist
- Annotations of key technologies (TensorFlow/Keras, OpenCV, Python).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & Preprocessing Layer
    subgraph "Data & Preprocessing"
        direction TB
        RawData["Raw .bin files"]:::storage
        prepare["prepare_data.py"]:::data
        datasets["datasets/"]:::storage
        augment["augment.py"]:::data
        dataLoader["data.py"]:::data
        dataDist["data_distiller.py"]:::data
        dataDrop["data_drop_top_k.py"]:::data
        tfdata["tf.data.Dataset"]:::data
    end

    %% Model Definition Layer
    subgraph "Model Definition"
        direction TB
        ghostV1["backbones/ghost_model.py"]:::model
        ghostV2["backbones/ghostv2.py"]:::model
        api["GhostFaceNets.py"]:::model
        registry["models.py"]:::model
    end

    %% Training Pipeline Layer
    subgraph "Training Pipeline"
        direction TB
        trainEngine["train.py (Train class)"]:::training
        losses["losses.py"]:::training
        callbacks["myCallbacks.py"]:::training
        checkpoints["checkpoints/"]:::storage
    end

    %% Evaluation Modules Layer
    subgraph "Evaluation Modules"
        direction TB
        evalCB["evals.py"]:::eval
        evalFolder["eval_folder.py"]:::eval
        IJB["IJB_evals.py"]:::eval
        subgraph "MegaFace Scripts"
            direction TB
            genMF["MegaFace/gen_megaface.py"]:::eval
            cleanMF["MegaFace/remove_noises.py"]:::eval
            runMF["MegaFace/run.sh"]:::eval
            nbMF["MegaFace/gen_megaface.ipynb"]:::eval
        end
    end

    %% Notebooks & Utilities Layer
    subgraph "Notebooks & Utilities"
        direction TB
        flops["Flops.ipynb"]:::util
        evalNB["evaluation.ipynb"]:::util
        trainNB1["training_scripts.ipynb"]:::util
        trainNB2["training_scripts_with_Bias.ipynb"]:::util
        plotParams["plot_parameters.py"]:::util
        video["video_test.py"]:::util
    end

    %% Data Flow Connections
    RawData -->|"convert .bin → folders"| prepare
    prepare -->|"outputs folders"| datasets
    datasets --> augment
    augment --> dataLoader
    dataLoader --> dataDist
    dataDist --> dataDrop
    dataDrop --> tfdata
    tfdata --> trainEngine

    %% Model to Training
    ghostV1 --> api
    ghostV2 --> api
    api --> registry
    registry --> trainEngine

    %% Training inner flow
    trainEngine -->|"uses losses"| losses
    trainEngine -->|"uses callbacks"| callbacks
    trainEngine -->|"fit() → .h5 models"| checkpoints

    %% Evaluation flow
    checkpoints --> evalCB
    checkpoints --> evalFolder
    checkpoints --> IJB
    checkpoints --> runMF

    %% Utilities Access
    checkpoints -.-> plotParams
    evalCB -.-> evalNB
    runMF -.-> nbMF
    trainEngine -.-> trainNB1
    trainEngine -.-> trainNB2
    trainEngine -.-> flops
    trainEngine -.-> video

    %% Click Events
    click prepare "https://github.com/hamadya/ghostfacenets/blob/main/prepare_data.py"
    click datasets "https://github.com/hamadya/ghostfacenets/tree/main/datasets/"
    click augment "https://github.com/hamadya/ghostfacenets/blob/main/augment.py"
    click dataLoader "https://github.com/hamadya/ghostfacenets/blob/main/data.py"
    click dataDist "https://github.com/hamadya/ghostfacenets/blob/main/data_distiller.py"
    click dataDrop "https://github.com/hamadya/ghostfacenets/blob/main/data_drop_top_k.py"
    click ghostV1 "https://github.com/hamadya/ghostfacenets/blob/main/backbones/ghost_model.py"
    click ghostV2 "https://github.com/hamadya/ghostfacenets/blob/main/backbones/ghostv2.py"
    click api "https://github.com/hamadya/ghostfacenets/blob/main/GhostFaceNets.py"
    click registry "https://github.com/hamadya/ghostfacenets/blob/main/models.py"
    click trainEngine "https://github.com/hamadya/ghostfacenets/blob/main/train.py"
    click losses "https://github.com/hamadya/ghostfacenets/blob/main/losses.py"
    click callbacks "https://github.com/hamadya/ghostfacenets/blob/main/myCallbacks.py"
    click checkpoints "https://github.com/hamadya/ghostfacenets/tree/main/checkpoints/"
    click evalCB "https://github.com/hamadya/ghostfacenets/blob/main/evals.py"
    click evalFolder "https://github.com/hamadya/ghostfacenets/blob/main/eval_folder.py"
    click IJB "https://github.com/hamadya/ghostfacenets/blob/main/IJB_evals.py"
    click genMF "https://github.com/hamadya/ghostfacenets/blob/main/MegaFace/gen_megaface.py"
    click cleanMF "https://github.com/hamadya/ghostfacenets/blob/main/MegaFace/remove_noises.py"
    click runMF "https://github.com/hamadya/ghostfacenets/blob/main/MegaFace/run.sh"
    click nbMF "https://github.com/hamadya/ghostfacenets/blob/main/MegaFace/gen_megaface.ipynb"
    click flops "https://github.com/hamadya/ghostfacenets/blob/main/Flops.ipynb"
    click evalNB "https://github.com/hamadya/ghostfacenets/blob/main/evaluation.ipynb"
    click trainNB1 "https://github.com/hamadya/ghostfacenets/blob/main/training_scripts.ipynb"
    click trainNB2 "https://github.com/hamadya/ghostfacenets/blob/main/training_scripts_with_Bias.ipynb"
    click plotParams "https://github.com/hamadya/ghostfacenets/blob/main/plot_parameters.py"
    click video "https://github.com/hamadya/ghostfacenets/blob/main/video_test.py"

    %% Styles
    classDef data fill:#D0E8FF,stroke:#009,stroke-width:1px
    classDef model fill:#D0FFD8,stroke:#090,stroke-width:1px
    classDef training fill:#FFE8B0,stroke:#E39,stroke-width:1px
    classDef eval fill:#E8D0FF,stroke:#909,stroke-width:1px
    classDef storage fill:#F0F0F0,stroke:#666,stroke-width:1px,stroke-dasharray: 2 2
    classDef util fill:#F5F5F5,stroke:#666,stroke-width:1px,stroke-dasharray: 5 5
```
