How erprogs/genconvit works
Project Type and Purpose
• This is a command-line, machine-learning pipeline for deepfake video detection.
• It comprises two independent neural networks (Autoencoder “ed” and Variational Autoencoder “vae”) plus a ConvNeXt-Swin hybrid, orchestrated for training and inference.
Major Architectural Components
A. Data Ingestion & Preprocessing
– Input video datasets (DFDC, FF++, DeepfakeTIMIT, Celeb-DF) under sample_train_data/ and sample_prediction_data/.
– dataset/loader.py: frame extraction (via decord), face detection (dlib, face_recognition, OpenCV), image normalization.
– Configuration (model/config.yaml, model/config.py).
B. Model Definition
– model/genconvit.py: builds the hybrid ConvNeXt + Swin Transformer stack.
– model/genconvit_ed.py: defines the Autoencoder branch (Network A).
– model/genconvit_vae.py: defines the Variational Autoencoder branch (Network B).
– model/model_embedder.py & model/pred_func.py: feature embedding and final classification logic.
C. Training Orchestration
– Entry point: train.py.
– Sub-modules: train/train_ed.py and train/train_vae.py.
– Responsibilities: parse CLI args (data path, model variant, epochs, batch size), launch training loops, compute losses, log metrics.
– Output: model weights and metrics saved to weight/.
D. Inference Pipeline
– Entry point: prediction.py.
– Loads trained weights (weight/genconvit_ed_*.pth, weight/genconvit_vae_*.pth) and configuration.
– Runs frame extraction + face detection → feature extraction → passes through ed and/or vae networks → aggregates predictions → outputs JSON or console results.
E. Results Aggregation
– result_all.py and JSON files in result/: collate per-dataset performance (accuracy, AUC, F1).
F. External Dependencies & Infrastructure
– PyTorch, torchvision, timm for model building.
– decord for video I/O.
– dlib, face_recognition, OpenCV for face detection/preprocessing.
– Python CLI environment, file-system storage for data, weights, and results.
Relationships & Data Flow
• Videos → Data Loader → Face crops → Model inputs
• CLI scripts (train.py, prediction.py) orchestrate each stage.
• Training writes weights to weight/; Inference reads from there.
• Metrics and predictions written to result/ and standard output.
Architectural Patterns & Design Principles
• Pipeline architecture: clear separation of data ingestion, model definition, training, and inference.
• Modular structure: each concern (loading, modeling, training, prediction) lives in its own folder.
• Config-driven: model hyperparameters in config.yaml.
• Dual-network ensemble pattern: ed and vae networks run independently or combined (GenConViT).