How vladhondru25/exddv works
Project type and purpose
• This is an open-source research pipeline for explainable deepfake detection in video. It orchestrates three separate vision-language model branches—BLIP-2 (via the LAVIS fork), LLaVA, and Phi3-Vision-Finetune—to extract embeddings, construct in-context prompts, run inference, and evaluate results.
File structure
• Top-level:
• ExDDV.csv – dataset annotations
• main.py, database.py, database_mysql.py – orchestration and optional result storage
• Notebooks (blip_incontext.ipynb, llava_incontext.ipynb, phi_incontext.ipynb) – interactive pipelines
• src/ with three subfolders:
– LAVIS (BLIP-2): feature extraction, retrieval, captioning, VQA, inference.py
– LLaVA: model builder, multimodal encoder, training & serve scripts
– Phi3-Vision-Finetune: training, merging LoRA weights, serve CLI/app
• Build & deployment: pyproject.toml, requirements.txt, run_scripts for each branch, conda/docker configs.
Key insights from README
• Pipeline stages: data preparation, embedding extraction, in-context prompt construction, deepfake analysis, evaluation and CSV/DB storage.
• Each model branch uses its own training scripts and code structure. Notebooks demonstrate in-context learning.
Main components
a. Data Layer
• Raw dataset: ExDDV.csv, source videos in data/…
• Frame extraction module (in database.py or inside src/utils)
• Optional database (MySQL) for storing embeddings & results
b. Embedding Extraction Services (three parallel modules)
• BLIP-2 (LAVIS/app & inference.py)
• LLaVA (llava/model & serve)
• Phi3-Vision (Phi3-Vision-Finetune/src/serve)
c. Retrieval & Prompt Builder
• Cosine similarity service (evaluate_general.py, utils.py)
• Top-k retrieval of training examples
• Prompt templates from each branch’s project/config
d. Model Inference & Analysis
• Each branch’s inference component consumes prompts + test frame embeddings
• Outputs deepfake detection explanations
e. Evaluation & Results
• CSV writer (main.py or evaluate_script.py)
• Database storage (database_mysql.py)
• Jupyter notebooks for interactive analysis
f. Orchestrator
• main.py coordinates end-to-end pipeline
• run_scripts/… shell scripts launch branch-specific training or evaluation
Relationships and interactions
• Data flows from CSV → Frame Extraction → Embedding Extraction → Embedding Store → Retrieval → Prompt Builder → Model Inference → Evaluation → Output
• The orchestrator (main.py or individual run_scripts) triggers each stage in sequence or in parallel per branch.
• Embedding store can be a local file system (NumPy/PyTorch tensors) or MySQL DB.
Architectural patterns & design principles
• Modular separation: three independent model pipelines under src/
• Pipeline pattern: clear sequential stages with well-defined inputs/outputs
• Reusability: common utilities (src/utils.py, common modules in LAVIS & LLaVA)
• Config-driven: YAML/JSON configs steer dataset paths, model parameters, prompt templates
Technologies, frameworks, libraries
• Python, PyTorch, HuggingFace Transformers
• LAVIS, LLaVA, Phi3-Vision-Finetune forks
• pandas for CSV I/O, scikit-learn or PyTorch for cosine similarity
• MySQL (optional) for persistent storage
• Jupyter Notebooks for interactive exploration