# erprogs/genconvit architecture

> Architecture diagram of the GitHub repository erprogs/genconvit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/erprogs/genconvit
- Repository: https://github.com/erprogs/genconvit

## Overview

##### Project Type and Purpose
- This is a command-line, machine-learning pipeline for deepfake video detection.
- It comprises two independent neural networks (Autoencoder “ed” and Variational Autoencoder “vae”) plus a ConvNeXt-Swin hybrid, orchestrated for training and inference.

##### Major Architectural Components
A. Data Ingestion & Preprocessing
– Input video datasets (DFDC, FF++, DeepfakeTIMIT, Celeb-DF) under `sample_train_data/` and `sample_prediction_data/`.
– `dataset/loader.py`: frame extraction (via decord), face detection (dlib, face_recognition, OpenCV), image normalization.
– Configuration (`model/config.yaml`, `model/config.py`).

B. Model Definition
– `model/genconvit.py`: builds the hybrid ConvNeXt + Swin Transformer stack.
– `model/genconvit_ed.py`: defines the Autoencoder branch (Network A).
– `model/genconvit_vae.py`: defines the Variational Autoencoder branch (Network B).
– `model/model_embedder.py` & `model/pred_func.py`: feature embedding and final classification logic.

C. Training Orchestration
– Entry point: `train.py`.
– Sub-modules: `train/train_ed.py` and `train/train_vae.py`.
– Responsibilities: parse CLI args (data path, model variant, epochs, batch size), launch training loops, compute losses, log metrics.
– Output: model weights and metrics saved to `weight/`.

D. Inference Pipeline
– Entry point: `prediction.py`.
– Loads trained weights (`weight/genconvit_ed_*.pth`, `weight/genconvit_vae_*.pth`) and configuration.
– Runs frame extraction + face detection → feature extraction → passes through ed and/or vae networks → aggregates predictions → outputs JSON or console results.

E. Results Aggregation
– `result_all.py` and JSON files in `result/`: collate per-dataset performance (accuracy, AUC, F1).

F. External Dependencies & Infrastructure
– PyTorch, torchvision, timm for model building.
– decord for video I/O.
– dlib, face_recognition, OpenCV for face detection/preprocessing.
– Python CLI environment, file-system storage for data, weights, and results.

##### Relationships & Data Flow
- Videos → Data Loader → Face crops → Model inputs
- CLI scripts (`train.py`, `prediction.py`) orchestrate each stage.
- Training writes weights to `weight/`; Inference reads from there.
- Metrics and predictions written to `result/` and standard output.

##### Architectural Patterns & Design Principles
- Pipeline architecture: clear separation of data ingestion, model definition, training, and inference.
- Modular structure: each concern (loading, modeling, training, prediction) lives in its own folder.
- Config-driven: model hyperparameters in `config.yaml`.
- Dual-network ensemble pattern: ed and vae networks run independently or combined (GenConViT).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Entry Points
    subgraph "CLI Entry Points"
        Train["train.py"]:::orchestrator
        Predict["prediction.py"]:::orchestrator
    end

    %% Data Storage
    subgraph "Data Storage"
        STData["sample_train_data/"]:::data
        SPData["sample_prediction_data/"]:::data
    end

    %% Preprocessing
    Loader["Face Detection & Frame Extraction\n(dataset/loader.py)"]:::compute

    %% Configuration
    ConfigMod["Configuration Module\n(model/config.py)"]:::compute
    ConfigYAML["Configuration File\n(model/config.yaml)"]:::compute

    %% Model Components
    subgraph "Model Definitions"
        Hybrid["Hybrid ConvNeXt-Swin\n(model/genconvit.py)"]:::compute
        ED["Autoencoder (ed)\n(model/genconvit_ed.py)"]:::compute
        VAE["Variational Autoencoder (vae)\n(model/genconvit_vae.py)"]:::compute
        Embedder["Feature Embedder & Classifier\n(model/model_embedder.py)"]:::compute
        PredFunc["Prediction Utilities\n(model/pred_func.py)"]:::compute
    end

    %% Storage for weights and results
    Weights["weight/"]:::storage
    Results["result/"]:::storage

    %% Results Aggregation Script
    ResultScript["result_all.py"]:::orchestrator

    %% Training Pipeline
    Train -->|"loads data"| STData
    Train -->|"uses loader"| Loader
    Train -->|"reads config"| ConfigMod
    Train -->|"reads config"| ConfigYAML
    Loader -->|"extracts frames"| ED
    Loader -->|"extracts frames"| VAE
    ED -->|"definition"| ED
    VAE -->|"definition"| VAE
    Train -->|"runs ED training"| TrainED
    Train -->|"runs VAE training"| TrainVAE
    TrainED["train/train_ed.py"]:::orchestrator -->|"saves weights"| Weights
    TrainVAE["train/train_vae.py"]:::orchestrator -->|"saves weights"| Weights

    %% Inference Pipeline
    Predict -->|"loads data"| SPData
    Predict -->|"uses loader"| Loader
    Predict -->|"loads weights"| Weights
    Loader -->|"preprocesses"| Hybrid
    Hybrid -->|"features"| Embedder
    Embedder -->|"classifies"| PredFunc
    PredFunc -->|"writes predictions"| Results
    ResultScript -->|"aggregates results"| Results

    %% Click Events
    click Loader "https://github.com/erprogs/genconvit/blob/main/dataset/loader.py"
    click STData "https://github.com/erprogs/genconvit/tree/main/sample_train_data/"
    click SPData "https://github.com/erprogs/genconvit/tree/main/sample_prediction_data/"
    click ConfigMod "https://github.com/erprogs/genconvit/blob/main/model/config.py"
    click ConfigYAML "https://github.com/erprogs/genconvit/blob/main/model/config.yaml"
    click Hybrid "https://github.com/erprogs/genconvit/blob/main/model/genconvit.py"
    click ED "https://github.com/erprogs/genconvit/blob/main/model/genconvit_ed.py"
    click VAE "https://github.com/erprogs/genconvit/blob/main/model/genconvit_vae.py"
    click Embedder "https://github.com/erprogs/genconvit/blob/main/model/model_embedder.py"
    click PredFunc "https://github.com/erprogs/genconvit/blob/main/model/pred_func.py"
    click Train "https://github.com/erprogs/genconvit/blob/main/train.py"
    click TrainED "https://github.com/erprogs/genconvit/blob/main/train/train_ed.py"
    click TrainVAE "https://github.com/erprogs/genconvit/blob/main/train/train_vae.py"
    click Weights "https://github.com/erprogs/genconvit/tree/main/weight/"
    click Predict "https://github.com/erprogs/genconvit/blob/main/prediction.py"
    click ResultScript "https://github.com/erprogs/genconvit/blob/main/result_all.py"
    click Results "https://github.com/erprogs/genconvit/tree/main/result/"

    %% Styles
    classDef data fill:#a6cee3,stroke:#1f78b4
    classDef compute fill:#b2df8a,stroke:#33a02c
    classDef orchestrator fill:#fdbf6f,stroke:#ff7f00
    classDef storage fill:#a6cee3,stroke:#1f78b4,stroke-dasharray: 5 5
```
