# fenilvadher/multimodal-deepfake-detection architecture

> Architecture diagram of the GitHub repository fenilvadher/multimodal-deepfake-detection, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fenilvadher/multimodal-deepfake-detection
- Repository: https://github.com/fenilvadher/multimodal-deepfake-detection

## Overview

##### Project type and purpose
- This is a monolithic, full-stack ML application built in Python with a Streamlit web UI.
- Its purpose is to detect deepfakes by analyzing three modalities—audio, image/video, and text—and fusing their predictions for a final verdict.

##### File structure
- Top-level files:
- DeepFakeDetection.py – the Streamlit entrypoint (UI + orchestration)
- requirements.txt – Python dependencies
- README.md – project overview
- Modality directories:
- Audio_DeepFake_Detection/ – notebooks, saved audio models (.pth)
- Image_DeepFake_Detection/ – notebooks, CSV metadata, training visuals
- Text_DeepFake_Detection/ – BERT/DeBERTa training and real-time detection notebooks, datasets
- Model artifacts and datasets are stored alongside code, indicating a functional decomposition by modality.

##### Key insights from the README
- Three specialist analysis engines: audio (MFCC + ML), image/video (ResNet18 CNN), text (BERT embeddings + logistic regression).
- A fusion module combines modality scores into a unified deepfake prediction.
- The Web App is implemented in Streamlit, which both serves the front end and orchestrates backend logic.

a. Main components
1. Streamlit Web UI (DeepFakeDetection.py)
2. Audio Analysis Module
3. Image/Video Analysis Module
4. Text Analysis Module
5. Multimodal Fusion Engine
6. Model Artifact Storage
7. (Optional) External News Verification API for real-time text validation
b. Relationships & data flow
- User uploads or streams media via Streamlit UI
- UI dispatches input to each specialist module
- Modules load their model artifact, preprocess input, run inference, and return a confidence score
- Fusion Engine aggregates the three scores and computes a final verdict
- UI presents the verdict and detailed scores back to the user
- (If used) Text module may call an external News Verification API to enrich real-time detection
c. Architectural patterns & design principles
- Functional decomposition: each modality is a self-contained processing pipeline
- Central orchestration: Streamlit script acts as the controller layer
- Separation of concerns: UI, preprocessing, inference, fusion
- Model artifacts are decoupled from code, enabling retraining or swapping models independently
d. Technologies & frameworks
- Streamlit (Web UI & orchestration)
- Python ML stack: PyTorch (audio & video models), scikit-learn (audio ML & logistic regression), Hugging Face Transformers (BERT/DeBERTa)
- MFCC feature extraction libraries for audio
- ResNet18 CNN for image frames
- Datasets in CSV/ZIP format, stored locally

##### Final advice
- Be exhaustive: represent each major notebook/demo component only if it embodies distinct functionality in your deployed pipeline.
- Critical data flows (uploads, inference calls, fusion, external API interactions).

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Presentation Layer"
        UI["Streamlit Web UI"]:::ui
    end

    subgraph "Processing Layer"
        AudioModule["Audio Analysis Module"]:::module
        ImageModule["Image/Video Analysis Module"]:::module
        TextModule["Text Analysis Module"]:::module
        DebertaValidation["DeBERTa Text Validation"]:::module
        RealTime["Real-Time Text Detection & News API"]:::module
        Fusion["Multimodal Fusion Engine"]:::module
    end

    subgraph "Data/Model Layer"
        AudioModel["Audio Model Artifact"]:::storage
        ImageDataset["Image Dataset"]:::storage
        ImageViz["Training Visualizations"]:::storage
        TextDataset["Text Dataset"]:::storage
    end

    ExternalAPI["News Verification API"]:::external

    UI -->|"audio waveform"| AudioModule
    UI -->|"image frames"| ImageModule
    UI -->|"text input"| TextModule

    AudioModule -->|"confidence score"| Fusion
    ImageModule -->|"confidence score"| Fusion
    TextModule -->|"confidence score"| Fusion
    Fusion -->|"final verdict"| UI

    TextModule --> DebertaValidation
    TextModule --> RealTime
    RealTime -->|"verification request"| ExternalAPI

    AudioModule --> AudioModel
    ImageModule --> ImageDataset
    ImageModule --> ImageViz
    TextModule --> TextDataset

    click UI "https://github.com/fenilvadher/multimodal-deepfake-detection/blob/main/DeepFakeDetection.py"
    click AudioModule "https://github.com/fenilvadher/multimodal-deepfake-detection/blob/main/Audio_DeepFake_Detection/Audio_deepfake_detection.ipynb"
    click AudioModel "https://github.com/fenilvadher/multimodal-deepfake-detection/blob/main/Audio_DeepFake_Detection/Model/audio_deepfake_model.pth"
    click ImageModule "https://github.com/fenilvadher/multimodal-deepfake-detection/blob/main/Image_DeepFake_Detection/image_ EfficientNetV2.ipynb"
    click ImageDataset "https://github.com/fenilvadher/multimodal-deepfake-detection/blob/main/Image_DeepFake_Detection/Datasets/dataset.csv"
    click ImageViz "https://github.com/fenilvadher/multimodal-deepfake-detection/tree/main/Image_DeepFake_Detection/training_visualizations"
    click TextModule "https://github.com/fenilvadher/multimodal-deepfake-detection/blob/main/Text_DeepFake_Detection/BERTModel_test.ipynb"
    click DebertaValidation "https://github.com/fenilvadher/multimodal-deepfake-detection/blob/main/Text_DeepFake_Detection/DeBERTa Model Validation Code.ipynb"
    click RealTime "https://github.com/fenilvadher/multimodal-deepfake-detection/blob/main/Text_DeepFake_Detection/Real-Time Detection System with News Verification API.ipynb"
    click TextDataset "https://github.com/fenilvadher/multimodal-deepfake-detection/blob/main/Text_DeepFake_Detection/Datasets/Fake-Real News Dataset.zip"
    click Fusion "https://github.com/fenilvadher/multimodal-deepfake-detection/blob/main/DeepFakeDetection.py"

    classDef ui fill:#D6EAF8,stroke:#2980B9,color:#154360
    classDef module fill:#D5F5E3,stroke:#27AE60,color:#145A32
    classDef storage fill:#FCF3CF,stroke:#F1C40F,color:#7D6608
    classDef external fill:#F4ECF7,stroke:#8E44AD,color:#4A235A
```
