# saksham-dev07/deepfake-forensics-with-explainable-ai architecture

> Architecture diagram of the GitHub repository saksham-dev07/deepfake-forensics-with-explainable-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saksham-dev07/deepfake-forensics-with-explainable-ai
- Repository: https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai

## Overview

1. This repo is split into two clear runtime surfaces: a Python backend under `backend/` and a React/Vite frontend under `frontend/`. The backend does all forensic inference; the frontend is a results dashboard and report viewer.

2. The backend entrypoint is `backend/main.py`, which is the API boundary exposed to the UI. Based on the README, it behaves like a FastAPI service that accepts media uploads, validates them, and returns both JSON results and downloadable PDF reports.

3. The main backend processing boundary is `backend/pipeline/`. This package contains the end-to-end forensic workflow: media preprocessing, per-signal analysis modules, score aggregation, explainability, and report generation. It is the core architectural center of the system.

4. `backend/pipeline/video_processor.py` is the ingestion layer for media. It appears to split uploaded video into frames and audio, enforce video duration constraints, and prepare inputs for the downstream analyzers. This is where the system turns a file upload into analysis-ready assets.

5. The analysis layer is multi-modal rather than single-model. The codebase includes separate modules for visual, temporal, audio, and metadata signals: face geometry, eye analysis, corneal reflection, lighting, CFA artifacts, ELA, frequency analysis, optical flow, rPPG, noise analysis, metadata analysis, and audio sync/voice spoofing. Each module contributes an independent anomaly score or feature vector.

6. Several heavyweight learned models are embedded as model wrappers in the pipeline, not trained at runtime. The key ones are an EfficientNet-based image detector, a SyncNet audio-visual alignment model, and a voice anti-spoofing model. Their weights live in `backend/weights/`, which makes model loading part of backend startup and inference, not frontend logic.

7. `backend/pipeline/ensemble_classifier.py` is the decision-making layer. It aggregates the outputs of all forensic modules into the final fake/real verdict and confidence profile. This is the main fusion boundary in the architecture: many specialized signals in, one ensemble score out.

8. Explainability and reporting are first-class outputs. `backend/pipeline/xai_explainer.py` generates interpretable evidence such as heatmaps or feature attributions, and `backend/pipeline/pdf_reporter.py` packages the findings into a court-style PDF report. These outputs are separate from the raw JSON response returned by the API.

9. The backend also has an offline training and experimentation sub-area. `backend/kaggle_scripts/` contains training scripts for the image, metadata/ensemble, and voice models, while `backend/test_prediction.py` suggests local inference validation. This training lane is distinct from the production inference path.

10. The frontend is a lightweight dashboard built with React and Vite. `frontend/src/App.jsx`, `ReportDashboard.jsx`, and `ModelsOverview.jsx` suggest the UI focuses on uploading media, displaying module-level results, and presenting the generated forensic report rather than performing analysis itself.

11. The main data flow is: browser upload -> API request to backend -> preprocessing and parallel forensic modules -> ensemble scoring -> XAI/report generation -> JSON and PDF returned to the UI. The frontend then renders the verdict and likely offers the PDF as a download or preview.

12. Operationally, the backend depends on local ML/CV runtime assets: PyTorch models, OpenCV-style frame processing, face-landmarker/face-detection files, and `ffmpeg` for media handling. It also enforces an API key and upload limits, which are important deployment boundaries because they shape how the service is called and protected.

13. For an architecture diagram, the key boxes are: client UI, API gateway, media preprocessing, parallel forensic analyzers, ensemble classifier, explainability/reporting, and model/weight storage. The cleanest boundary separation is between interactive presentation in `frontend/` and compute-heavy inference in `backend/`.

Main entry points: [`backend/pipeline/__init__.py`](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/__init__.py) (Pipeline), [`backend/pipeline/ensemble_classifier.py`](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/ensemble_classifier.py) (Fusion), [`backend/main.py`](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/main.py) (API), [`backend/weights`](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/tree/main/backend/weights) (Weights)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Frontend"]
  node_ui["UI<br/>React app<br/>[App.jsx]"]
  node_dashboard["Report view<br/>React component"]
  node_models_ui["Models view<br/>React component<br/>[ModelsOverview.jsx]"]
end

subgraph group_backend["Backend"]
  node_api["API<br/>[main.py]"]
  node_processor["Video prep<br/>media ingestion<br/>[video_processor.py]"]
  node_pipeline["Pipeline<br/>forensic workflow<br/>[__init__.py]"]
  node_face["Face signals<br/>visual analysis<br/>[face_geometry.py]"]
  node_image_artifacts["Image cues<br/>artifact analysis"]
  node_motion["Motion cues<br/>temporal analysis<br/>[optical_flow.py]"]
  node_audio["Audio cues<br/>audio analysis<br/>[audio_sync.py]"]
  node_metadata["Metadata<br/>file analysis"]
  node_ensemble["Fusion<br/>ensemble classifier"]
  node_xai["Explainability<br/>XAI output<br/>[xai_explainer.py]"]
  node_report["PDF report<br/>report generator<br/>[pdf_reporter.py]"]
  node_syncnet["SyncNet<br/>AV model<br/>[SyncNetModel.py]"]
  node_voice_model["Voice model<br/>spoof model<br/>[voice_model.py]"]
end

subgraph group_training["Offline training"]
  node_train_scripts["Training scripts<br/>offline training"]
end

subgraph group_assets["Model assets"]
  node_weights[("Weights<br/>model assets")]
end

node_ui -->|"uploads"| node_api
node_dashboard -->|"fetches results"| node_api
node_models_ui -->|"shows signals"| node_api
node_api -->|"ingests"| node_processor
node_processor -->|"hands off"| node_pipeline
node_pipeline -->|"routes"| node_face
node_pipeline -->|"routes"| node_image_artifacts
node_pipeline -->|"routes"| node_motion
node_pipeline -->|"routes"| node_audio
node_pipeline -->|"routes"| node_metadata
node_face -->|"scores"| node_ensemble
node_image_artifacts -->|"scores"| node_ensemble
node_motion -->|"scores"| node_ensemble
node_audio -->|"scores"| node_ensemble
node_metadata -->|"scores"| node_ensemble
node_syncnet -->|"powers"| node_audio
node_voice_model -->|"powers"| node_audio
node_weights -.->|"loads"| node_syncnet
node_weights -.->|"loads"| node_voice_model
node_weights -.->|"loads"| node_ensemble
node_pipeline -->|"explains"| node_xai
node_ensemble -->|"exposes"| node_xai
node_pipeline -->|"packages"| node_report
node_ensemble -->|"summarizes"| node_report
node_train_scripts -.->|"produces"| node_weights
node_api -->|"returns"| node_report

click node_ui "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/frontend/src/App.jsx"
click node_dashboard "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/frontend/src/components/ReportDashboard.jsx"
click node_models_ui "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/frontend/src/components/ModelsOverview.jsx"
click node_api "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/main.py"
click node_processor "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/video_processor.py"
click node_pipeline "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/__init__.py"
click node_face "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/face_geometry.py"
click node_image_artifacts "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/lighting_analysis.py"
click node_motion "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/optical_flow.py"
click node_audio "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/audio_sync.py"
click node_metadata "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/metadata_analysis.py"
click node_ensemble "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/ensemble_classifier.py"
click node_xai "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/xai_explainer.py"
click node_report "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/pdf_reporter.py"
click node_syncnet "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/SyncNetModel.py"
click node_voice_model "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/voice_model.py"
click node_weights "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/tree/main/backend/weights"
click node_train_scripts "https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/kaggle_scripts/kaggle_efficientnet_training.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui,node_dashboard,node_models_ui toneBlue
class node_api,node_processor,node_pipeline,node_face,node_image_artifacts,node_motion,node_audio,node_metadata,node_ensemble,node_xai,node_report,node_syncnet,node_voice_model toneAmber
class node_train_scripts toneMint
class node_weights toneRose
```

## Components

### Frontend

React/Vite dashboard and report viewer

- [UI](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/frontend/src/App.jsx): React app · Browser dashboard for uploads, verdicts, and reports · `frontend/src/App.jsx`
- [Report view](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/frontend/src/components/ReportDashboard.jsx): React component · Renders analysis results and generated report details · `frontend/src/components/ReportDashboard.jsx`
- [Models view](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/frontend/src/components/ModelsOverview.jsx): React component · Summarizes available forensic models and signals · `frontend/src/components/ModelsOverview.jsx`

### Backend

Python API and forensic inference pipeline

- [API](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/main.py): FastAPI entrypoint · Upload and inference boundary exposed to the UI · `backend/main.py`
- [Video prep](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/video_processor.py): media ingestion · Splits uploads into frames/audio and validates inputs · `backend/pipeline/video_processor.py`
- [Pipeline](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/__init__.py): forensic workflow · Coordinates preprocessing, analysis, fusion, and outputs · `backend/pipeline/__init__.py`
- [Face signals](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/face_geometry.py): visual analysis · Face geometry, eyes, corneal reflection, and landmark-based cues · `backend/pipeline/face_geometry.py`
- [Image cues](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/lighting_analysis.py): artifact analysis · Lighting, color, CFA, ELA, frequency, and noise checks · `backend/pipeline/lighting_analysis.py`
- [Motion cues](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/optical_flow.py): temporal analysis · Optical flow and rPPG-based temporal consistency signals · `backend/pipeline/optical_flow.py`
- [Audio cues](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/audio_sync.py): audio analysis · Audio sync and voice spoofing detection · `backend/pipeline/audio_sync.py`
- [Metadata](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/metadata_analysis.py): file analysis · Container and EXIF-style metadata anomaly checks · `backend/pipeline/metadata_analysis.py`
- [Fusion](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/ensemble_classifier.py): ensemble classifier · Combines module scores into final real/fake verdict · `backend/pipeline/ensemble_classifier.py`
- [Explainability](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/xai_explainer.py): XAI output · Generates heatmaps and feature attributions · `backend/pipeline/xai_explainer.py`
- [PDF report](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/pdf_reporter.py): report generator · Packages findings into a downloadable court-style PDF · `backend/pipeline/pdf_reporter.py`
- [SyncNet](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/SyncNetModel.py): AV model · Audio-visual alignment model wrapper · `backend/pipeline/SyncNetModel.py`
- [Voice model](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/pipeline/voice_model.py): spoof model · Voice anti-spoofing inference wrapper · `backend/pipeline/voice_model.py`

### Offline training

Kaggle-oriented model training scripts

- [Training scripts](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/blob/main/backend/kaggle_scripts/kaggle_efficientnet_training.py): offline training · Kaggle scripts for image, metadata, and voice models · `backend/kaggle_scripts/kaggle_efficientnet_training.py`

### Model assets

Stored weights and runtime ML artifacts

- [Weights](https://github.com/saksham-dev07/deepfake-forensics-with-explainable-ai/tree/main/backend/weights): model assets · Stored learned weights and detector artifacts · `backend/weights`

## Connections

- UI → API: uploads · Sends media for forensic analysis
- Report view → API: fetches results · Reads JSON verdicts and report metadata
- Models view → API: shows signals · Displays available module-level outputs
- API → Video prep: ingests · Validates and prepares uploaded media
- Video prep → Pipeline: hands off · Passes analysis-ready frames and audio
- Pipeline → Face signals: routes · Executes face and landmark-based checks
- Pipeline → Image cues: routes · Executes artifact and image consistency checks
- Pipeline → Motion cues: routes · Executes temporal and motion-based checks
- Pipeline → Audio cues: routes · Executes audio consistency checks
- Pipeline → Metadata: routes · Executes file metadata analysis
- Face signals → Fusion: scores · Contributes anomaly evidence to fusion
- Image cues → Fusion: scores · Contributes artifact evidence to fusion
- Motion cues → Fusion: scores · Contributes temporal evidence to fusion
- Audio cues → Fusion: scores · Contributes audio evidence to fusion
- Metadata → Fusion: scores · Contributes metadata evidence to fusion
- SyncNet → Audio cues: powers · Provides audio-visual alignment inference
- Voice model → Audio cues: powers · Provides spoofing inference
- Weights → SyncNet: loads · Stores model parameters and detector artifacts
- Weights → Voice model: loads · Stores model parameters and detector artifacts
- Weights → Fusion: loads · Stores fusion model weights
- Pipeline → Explainability: explains · Generates interpretable evidence from inference
- Fusion → Explainability: exposes · Provides final decision signals for attribution
- Pipeline → PDF report: packages · Builds downloadable PDF report output
- Fusion → PDF report: summarizes · Supplies verdict and confidence profile
- Training scripts → Weights: produces · Offline training writes learned model artifacts
- API → PDF report: returns · Serves report download alongside JSON response
