How fenilvadher/multimodal-deepfake-detection works
Project type and purpose
• This is a monolithic, full-stack ML application built in Python with a Streamlit web UI.
• Its purpose is to detect deepfakes by analyzing three modalities—audio, image/video, and text—and fusing their predictions for a final verdict.
File structure
• Top-level files:
• DeepFakeDetection.py – the Streamlit entrypoint (UI + orchestration)
• requirements.txt – Python dependencies
• README.md – project overview
• Modality directories:
• Audio_DeepFake_Detection/ – notebooks, saved audio models (.pth)
• Image_DeepFake_Detection/ – notebooks, CSV metadata, training visuals
• Text_DeepFake_Detection/ – BERT/DeBERTa training and real-time detection notebooks, datasets
• Model artifacts and datasets are stored alongside code, indicating a functional decomposition by modality.
Key insights from the README
• Three specialist analysis engines: audio (MFCC + ML), image/video (ResNet18 CNN), text (BERT embeddings + logistic regression).
• A fusion module combines modality scores into a unified deepfake prediction.
• The Web App is implemented in Streamlit, which both serves the front end and orchestrates backend logic.
a. Main components
1. Streamlit Web UI (DeepFakeDetection.py)
2. Audio Analysis Module
3. Image/Video Analysis Module
4. Text Analysis Module
5. Multimodal Fusion Engine
6. Model Artifact Storage
7. (Optional) External News Verification API for real-time text validation
b. Relationships & data flow
• User uploads or streams media via Streamlit UI
• UI dispatches input to each specialist module
• Modules load their model artifact, preprocess input, run inference, and return a confidence score
• Fusion Engine aggregates the three scores and computes a final verdict
• UI presents the verdict and detailed scores back to the user
• (If used) Text module may call an external News Verification API to enrich real-time detection
c. Architectural patterns & design principles
• Functional decomposition: each modality is a self-contained processing pipeline
• Central orchestration: Streamlit script acts as the controller layer
• Separation of concerns: UI, preprocessing, inference, fusion
• Model artifacts are decoupled from code, enabling retraining or swapping models independently
d. Technologies & frameworks
• Streamlit (Web UI & orchestration)
• Python ML stack: PyTorch (audio & video models), scikit-learn (audio ML & logistic regression), Hugging Face Transformers (BERT/DeBERTa)
• MFCC feature extraction libraries for audio
• ResNet18 CNN for image frames
• Datasets in CSV/ZIP format, stored locally
Final advice
• Be exhaustive: represent each major notebook/demo component only if it embodies distinct functionality in your deployed pipeline.
• Critical data flows (uploads, inference calls, fusion, external API interactions).