How saksham-dev07/deepfake-forensics-with-explainable-ai works
1. This repo is split into two clear runtime surfaces: a Python backend under backend/ and a React/Vite frontend under frontend/. The backend does all forensic inference; the frontend is a results dashboard and report viewer.
2. The backend entrypoint is backend/main.py, which is the API boundary exposed to the UI. Based on the README, it behaves like a FastAPI service that accepts media uploads, validates them, and returns both JSON results and downloadable PDF reports.
3. The main backend processing boundary is backend/pipeline/. This package contains the end-to-end forensic workflow: media preprocessing, per-signal analysis modules, score aggregation, explainability, and report generation. It is the core architectural center of the system.
4. backend/pipeline/video_processor.py is the ingestion layer for media. It appears to split uploaded video into frames and audio, enforce video duration constraints, and prepare inputs for the downstream analyzers. This is where the system turns a file upload into analysis-ready assets.
5. The analysis layer is multi-modal rather than single-model. The codebase includes separate modules for visual, temporal, audio, and metadata signals: face geometry, eye analysis, corneal reflection, lighting, CFA artifacts, ELA, frequency analysis, optical flow, rPPG, noise analysis, metadata analysis, and audio sync/voice spoofing. Each module contributes an independent anomaly score or feature vector.
6. Several heavyweight learned models are embedded as model wrappers in the pipeline, not trained at runtime. The key ones are an EfficientNet-based image detector, a SyncNet audio-visual alignment model, and a voice anti-spoofing model. Their weights live in backend/weights/, which makes model loading part of backend startup and inference, not frontend logic.
7. backend/pipeline/ensemble_classifier.py is the decision-making layer. It aggregates the outputs of all forensic modules into the final fake/real verdict and confidence profile. This is the main fusion boundary in the architecture: many specialized signals in, one ensemble score out.
8. Explainability and reporting are first-class outputs. backend/pipeline/xai_explainer.py generates interpretable evidence such as heatmaps or feature attributions, and backend/pipeline/pdf_reporter.py packages the findings into a court-style PDF report. These outputs are separate from the raw JSON response returned by the API.
9. The backend also has an offline training and experimentation sub-area. backend/kaggle_scripts/ contains training scripts for the image, metadata/ensemble, and voice models, while backend/test_prediction.py suggests local inference validation. This training lane is distinct from the production inference path.
10. The frontend is a lightweight dashboard built with React and Vite. frontend/src/App.jsx, ReportDashboard.jsx, and ModelsOverview.jsx suggest the UI focuses on uploading media, displaying module-level results, and presenting the generated forensic report rather than performing analysis itself.
11. The main data flow is: browser upload -> API request to backend -> preprocessing and parallel forensic modules -> ensemble scoring -> XAI/report generation -> JSON and PDF returned to the UI. The frontend then renders the verdict and likely offers the PDF as a download or preview.
12. Operationally, the backend depends on local ML/CV runtime assets: PyTorch models, OpenCV-style frame processing, face-landmarker/face-detection files, and ffmpeg for media handling. It also enforces an API key and upload limits, which are important deployment boundaries because they shape how the service is called and protected.
13. For an architecture diagram, the key boxes are: client UI, API gateway, media preprocessing, parallel forensic analyzers, ensemble classifier, explainability/reporting, and model/weight storage. The cleanest boundary separation is between interactive presentation in frontend/ and compute-heavy inference in backend/.
Main entry points: __init__.py Pipeline · ensemble_classifier.py Fusion · main.py API · weights Weights
Components 18
Frontend
React/Vite dashboard and report viewer
- UI · Browser dashboard for uploads, verdicts, and reports
frontend/src/App.jsx - Report view · Renders analysis results and generated report details
frontend/src/components/ReportDashboard.jsx - Models view · Summarizes available forensic models and signals
frontend/src/components/ModelsOverview.jsx
Backend
Python API and forensic inference pipeline
- API · Upload and inference boundary exposed to the UI
backend/main.py - Video prep · Splits uploads into frames/audio and validates inputs
backend/pipeline/video_processor.py - Pipeline · Coordinates preprocessing, analysis, fusion, and outputs
backend/pipeline/__init__.py - Face signals · Face geometry, eyes, corneal reflection, and landmark-based cues
backend/pipeline/face_geometry.py - Image cues · Lighting, color, CFA, ELA, frequency, and noise checks
backend/pipeline/lighting_analysis.py - Motion cues · Optical flow and rPPG-based temporal consistency signals
backend/pipeline/optical_flow.py - Audio cues · Audio sync and voice spoofing detection
backend/pipeline/audio_sync.py - Metadata · Container and EXIF-style metadata anomaly checks
backend/pipeline/metadata_analysis.py - Fusion · Combines module scores into final real/fake verdict
backend/pipeline/ensemble_classifier.py - Explainability · Generates heatmaps and feature attributions
backend/pipeline/xai_explainer.py - PDF report · Packages findings into a downloadable court-style PDF
backend/pipeline/pdf_reporter.py - SyncNet · Audio-visual alignment model wrapper
backend/pipeline/SyncNetModel.py - Voice model · Voice anti-spoofing inference wrapper
backend/pipeline/voice_model.py
Offline training
Kaggle-oriented model training scripts
- Training scripts · Kaggle scripts for image, metadata, and voice models
backend/kaggle_scripts/kaggle_efficientnet_training.py
Model assets
Stored weights and runtime ML artifacts
- Weights · Stored learned weights and detector artifacts
backend/weights
Connections 26
- UI → API: uploads · Sends media for forensic analysis
- Report view → API: fetches results · Reads JSON verdicts and report metadata
- Models view → API: shows signals · Displays available module-level outputs
- API → Video prep: ingests · Validates and prepares uploaded media
- Video prep → Pipeline: hands off · Passes analysis-ready frames and audio
- Pipeline → Face signals: routes · Executes face and landmark-based checks
- Pipeline → Image cues: routes · Executes artifact and image consistency checks
- Pipeline → Motion cues: routes · Executes temporal and motion-based checks
- Pipeline → Audio cues: routes · Executes audio consistency checks
- Pipeline → Metadata: routes · Executes file metadata analysis
- Face signals → Fusion: scores · Contributes anomaly evidence to fusion
- Image cues → Fusion: scores · Contributes artifact evidence to fusion
- Motion cues → Fusion: scores · Contributes temporal evidence to fusion
- Audio cues → Fusion: scores · Contributes audio evidence to fusion
- Metadata → Fusion: scores · Contributes metadata evidence to fusion
- SyncNet → Audio cues: powers · Provides audio-visual alignment inference
- Voice model → Audio cues: powers · Provides spoofing inference
- Weights → SyncNet: loads · Stores model parameters and detector artifacts
- Weights → Voice model: loads · Stores model parameters and detector artifacts
- Weights → Fusion: loads · Stores fusion model weights
- Pipeline → Explainability: explains · Generates interpretable evidence from inference
- Fusion → Explainability: exposes · Provides final decision signals for attribution
- Pipeline → PDF report: packages · Builds downloadable PDF report output
- Fusion → PDF report: summarizes · Supplies verdict and confidence profile
- Training scripts → Weights: produces · Offline training writes learned model artifacts
- API → PDF report: returns · Serves report download alongside JSON response