# selimsef/dfdc_deepfake_challenge architecture

> Architecture diagram of the GitHub repository selimsef/dfdc_deepfake_challenge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/selimsef/dfdc_deepfake_challenge
- Repository: https://github.com/selimsef/dfdc_deepfake_challenge

## Overview

##### Project Type and Purpose
- The project is a deepfake detection solution built for the Kaggle DeepFake Detection Challenge. It focuses on a frame-by-frame classification approach to determine if a video is manipulated.
- The README shows that the system includes multiple processing steps (face detection, crop extraction, landmark generation, and difference mask creation) before training classification models (using EfficientNet encoders, primarily B7) and finally carrying out inference via ensemble predictions.

##### Analyzing the File Structure for Architectural Components
- Top-level components include:
- Dockerfile: Containerizing the entire solution.
- Preprocessing folder (with scripts such as detect_original_faces.py, extract_crops.py, generate_landmarks.py, generate_diffs.py, generate_folds.py): This clearly indicates a dedicated data preparation pipeline.
- Training folder (which further has subfolders like datasets, pipelines, tools, transforms, zoo): This separates model training components (data handling, loss calculations, augmentation transforms, and model definitions) from preprocessing.
- Inference scripts such as predict_folder.py and predict_submission.sh: They represent the prediction engine used for scoring videos.
- Supporting files such as configurations (configs folder), logging (logs folder), and utility scripts (kernel_utils, plot_loss) that aid with the overall orchestration.
- The structure suggests a modular design separating concerns between data preparation, model training (with augmentations and data pipelines), and inference.

##### Key Components
a. Preprocessing Module
- Input: Raw video dataset (the DFDC challenge dataset).
- Components:
- Face detection (using MTCNN as implemented in preprocessing/face_detector.py and detect_original_faces.py).
- Crop extraction (via preprocessing/extract_crops.py).
- Landmark extraction (preprocessing/generate_landmarks.py).
- Difference mask generation (preprocessing/generate_diffs.py).
- Fold generation for cross-validation (preprocessing/generate_folds.py).
- Output: Processed data including face crops, landmarks, SSIM difference masks, and fold information.

b. Training Module
- Components:
- Datasets and pipelines: Found under training/datasets and training/pipelines, which prepare and feed data into the training process.
- Data augmentation: Heavy augmentations implemented with Albumentations (shown in the README and in training/transforms).
- Model zoo: Under training/zoo, which contains different deep learning models—a classifier (B7) and a Unet reference, showing an ensemble strategy.
- Training tools and utilities: Configurations, loss functions, schedulers, and supporting utilities that tune training.
- Flow: Processed data -> augmentation -> training pipelines -> model training -> saving checkpoints & logs.

c. Inference Module
- Components:
- predict_folder.py and predict_submission.sh: Scripts to run inference on new datasets.
- Ensemble strategy: Aggregates predictions from multiple frames (with special handling such as the “confident_strategy” to average predictions).
- Flow: Load trained weights -> process video frames -> generate predictions -> ensemble output.

d. Supporting Infrastructure
- Docker Container: Dockerfile encapsulates the environment and dependencies (e.g., Nvidia-docker support).
- Configurations: The configs directory (with b5.json and b7.json) which likely feeds model/hyperparameter settings.
- External dependencies: Pretrained model weights (download_weights.sh downloads models to weights folder) and relevant libraries (Albumentations, EfficientNet implementations, etc.).

##### Relationships and Data Flow
- These outputs form the training dataset, which is fed into data augmentation pipelines and then into the training models.
- Trained models (or checkpoints) are stored and later loaded during inference.
- Inference scripts access the saved weights and run predictions over input videos, applying the ensemble heuristic on the frame-level results.
- Docker encapsulates and standardizes the runtime environment throughout these processes.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Deployment/Containerization Layer
    D1["Docker Container"]:::deployment

    %% External Input
    RVD["Raw Video Dataset"]:::external

    %% Data Ingestion/Preprocessing Layer
    subgraph "Data Ingestion/Preprocessing Layer"
        P1["Face Detection (face_detector.py)"]:::preprocessing
        P2["Face Detection (detect_original_faces.py)"]:::preprocessing
        P3["Crop Extraction (extract_crops.py)"]:::preprocessing
        P4["Landmark Extraction (generate_landmarks.py)"]:::preprocessing
        P5["Diff Mask Generation (generate_diffs.py)"]:::preprocessing
        P6["Fold Generation (generate_folds.py)"]:::preprocessing
        PD["Processed Data"]:::preprocessing
    end

    %% Model Training Layer
    subgraph "Model Training Layer"
        T1["Classifier Dataset (classifier_dataset.py)"]:::training
        T2["Validation Set (validation_set.py)"]:::training
        T4["Data Augmentation (albu.py)"]:::training
        T3["Training Pipeline (train_classifier.py)"]:::training
        T7["Loss Functions (losses.py)"]:::training
        T8["Config (config.py)"]:::training
        T9["Schedulers (schedulers.py)"]:::training
        T10["Utils (utils.py)"]:::training
        T5["Classifier Models (classifiers.py)"]:::training
        T6["Unet Model (unet.py)"]:::training
        LM["Trained Models"]:::training
    end

    %% Inference/Prediction Layer
    subgraph "Inference/Prediction Layer"
        I1["Predict Folder (predict_folder.py)"]:::inference
        I2["Predict Submission (predict_submission.sh)"]:::inference
        IE["Inference Engine"]:::inference
        FP["Final Predictions"]:::inference
    end

    %% Supporting Infrastructure
    subgraph "Supporting Infrastructure"
        S1a["Config b5 (b5.json)"]:::support
        S1b["Config b7 (b7.json)"]:::support
        S2["Download Weights (download_weights.sh)"]:::support
        S3["Weights Directory"]:::support
    end

    %% Connections from Deployment
    D1 --> RVD
    D1 --- P1
    D1 --- T3
    D1 --- IE

    %% Data Ingestion/Preprocessing Flow
    RVD --> P1
    P1 --> P2
    P2 --> P3
    P3 --> P4
    P4 --> P5
    P5 --> P6
    P6 --> PD

    %% Training Flow
    PD --> T1
    PD --> T2
    T1 --> T3
    T2 --> T3
    T4 --- T3
    T7 -.-> T3
    T8 -.-> T3
    T9 -.-> T3
    T10 -.-> T3
    T3 --> T5
    T3 --> T6
    T5 --> LM
    T6 --> LM
    S1a --- T3
    S1b --- T3
    S2 --- LM
    S3 --- LM

    %% Inference Flow
    LM --> IE
    I1 --> IE
    I2 --> IE
    IE --> FP

    %% Style Classes
    classDef deployment fill:#D3D3D3,stroke:#000,stroke-width:2px;
    classDef external fill:#F5DEB3,stroke:#000,stroke-width:2px;
    classDef preprocessing fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef training fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef inference fill:#FFFF99,stroke:#000,stroke-width:2px;
    classDef support fill:#FFB6C1,stroke:#000,stroke-width:2px;

    %% Click Events
    click D1 "https://github.com/selimsef/dfdc_deepfake_challenge/tree/master/Dockerfile"
    click P1 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/preprocessing/face_detector.py"
    click P2 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/preprocessing/detect_original_faces.py"
    click P3 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/preprocessing/extract_crops.py"
    click P4 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/preprocessing/generate_landmarks.py"
    click P5 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/preprocessing/generate_diffs.py"
    click P6 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/preprocessing/generate_folds.py"
    click T1 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/training/datasets/classifier_dataset.py"
    click T2 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/training/datasets/validation_set.py"
    click T3 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/training/pipelines/train_classifier.py"
    click T4 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/training/transforms/albu.py"
    click T5 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/training/zoo/classifiers.py"
    click T6 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/training/zoo/unet.py"
    click T7 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/training/losses.py"
    click T8 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/training/tools/config.py"
    click T9 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/training/tools/schedulers.py"
    click T10 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/training/tools/utils.py"
    click I1 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/predict_folder.py"
    click I2 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/predict_submission.sh"
    click S1a "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/configs/b5.json"
    click S1b "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/configs/b7.json"
    click S2 "https://github.com/selimsef/dfdc_deepfake_challenge/blob/master/download_weights.sh"
    click S3 "https://github.com/selimsef/dfdc_deepfake_challenge/tree/master/weights"
```
