# vladhondru25/exddv architecture

> Architecture diagram of the GitHub repository vladhondru25/exddv, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vladhondru25/exddv
- Repository: https://github.com/vladhondru25/exddv

## Overview

##### Project type and purpose
- This is an open-source research pipeline for explainable deepfake detection in video. It orchestrates three separate vision-language model branches—BLIP-2 (via the LAVIS fork), LLaVA, and Phi3-Vision-Finetune—to extract embeddings, construct in-context prompts, run inference, and evaluate results.

##### File structure
- Top-level:
- ExDDV.csv – dataset annotations
- main.py, database.py, database_mysql.py – orchestration and optional result storage
- Notebooks (blip_incontext.ipynb, llava_incontext.ipynb, phi_incontext.ipynb) – interactive pipelines
- src/ with three subfolders:
– LAVIS (BLIP-2): feature extraction, retrieval, captioning, VQA, inference.py
– LLaVA: model builder, multimodal encoder, training & serve scripts
– Phi3-Vision-Finetune: training, merging LoRA weights, serve CLI/app
- Build & deployment: pyproject.toml, requirements.txt, run_scripts for each branch, conda/docker configs.

##### Key insights from README
- Pipeline stages: data preparation, embedding extraction, in-context prompt construction, deepfake analysis, evaluation and CSV/DB storage.
- Each model branch uses its own training scripts and code structure. Notebooks demonstrate in-context learning.

##### Main components
a. Data Layer
- Raw dataset: ExDDV.csv, source videos in data/…
- Frame extraction module (in database.py or inside src/utils)
- Optional database (MySQL) for storing embeddings & results
b. Embedding Extraction Services (three parallel modules)
- BLIP-2 (LAVIS/app & inference.py)
- LLaVA (llava/model & serve)
- Phi3-Vision (Phi3-Vision-Finetune/src/serve)
c. Retrieval & Prompt Builder
- Cosine similarity service (evaluate_general.py, utils.py)
- Top-k retrieval of training examples
- Prompt templates from each branch’s project/config
d. Model Inference & Analysis
- Each branch’s inference component consumes prompts + test frame embeddings
- Outputs deepfake detection explanations
e. Evaluation & Results
- CSV writer (main.py or evaluate_script.py)
- Database storage (database_mysql.py)
- Jupyter notebooks for interactive analysis
f. Orchestrator
- main.py coordinates end-to-end pipeline
- run_scripts/… shell scripts launch branch-specific training or evaluation

##### Relationships and interactions
- Data flows from CSV → Frame Extraction → Embedding Extraction → Embedding Store → Retrieval → Prompt Builder → Model Inference → Evaluation → Output
- The orchestrator (main.py or individual run_scripts) triggers each stage in sequence or in parallel per branch.
- Embedding store can be a local file system (NumPy/PyTorch tensors) or MySQL DB.

##### Architectural patterns & design principles
- Modular separation: three independent model pipelines under src/
- Pipeline pattern: clear sequential stages with well-defined inputs/outputs
- Reusability: common utilities (src/utils.py, common modules in LAVIS & LLaVA)
- Config-driven: YAML/JSON configs steer dataset paths, model parameters, prompt templates

##### Technologies, frameworks, libraries
- Python, PyTorch, HuggingFace Transformers
- LAVIS, LLaVA, Phi3-Vision-Finetune forks
- pandas for CSV I/O, scikit-learn or PyTorch for cosine similarity
- MySQL (optional) for persistent storage
- Jupyter Notebooks for interactive exploration

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Layer"
        Raw["Raw Dataset\nExDDV.csv"]:::data
        FrameExt["Frame Extraction\nmodule"]:::utils
        EmbStore["Embedding Store\n(Local FS)"]:::storage
        MySQLDB["MySQL DB (optional)"]:::storage
    end

    Raw --> FrameExt
    FrameExt --> EmbStore
    FrameExt -.-> MySQLDB

    subgraph "Embedding Extraction Services"
        subgraph "BLIP-2 (LAVIS)"
            LavisApp["Feature Extraction & Captioning\n(src/LAVIS/app/)"]:::lavis
            LavisInf["BLIP-2 Inference\n(inference.py)"]:::lavis
            LavisEvalGen["Cosine & Retrieval\n(evaluate_general.py)"]:::lavis
            LavisEval["Evaluation Scripts\n(evaluate_script.py)"]:::lavis
        end
        subgraph "LLaVA"
            Llavabuilder["Model Builder\n(builder.py)"]:::llava
            LlamaEnc["Multimodal Encoder\n(multimodal_encoder/)"]:::llava
            LlavaScripts["Training Scripts\n(src/LLaVA/scripts/)"]:::llava
            LlavaServeCLI["Serve CLI\n(cli.py)"]:::llava
            LlavaController["Serve Controller\n(controller.py)"]:::llava
        end
        subgraph "Phi3-Vision-Finetune"
            PhiTrain["Training Pipeline\n(train.py)"]:::phi3
            PhiMerge["LoRA Merge\n(merge_lora_weights.py)"]:::phi3
            PhiServeInf["Serve Inference\n(inference.py)"]:::phi3
            PhiServeCLI["Serve CLI\n(cli.py)"]:::phi3
            PhiServeApp["Serve App\n(app.py)"]:::phi3
        end
    end

    EmbStore --> LavisApp
    EmbStore --> Llavabuilder
    EmbStore --> PhiTrain

    LavisApp --> LavisInf
    Llavabuilder --> LlamaEnc
    LlamaEnc --> LlavaServeCLI
    PhiTrain --> PhiMerge

    subgraph "Retrieval & Prompt Builder"
        Cosine["Cosine Similarity & Top-k\n(evaluate_general.py)"]:::builder
        PromptUtils["Prompt Utils\n(app/utils.py)"]:::builder
        PromptConfig["Prompt Templates\n(pretrain_stage2_xai.yaml)"]:::builder
    end

    EmbStore --> Cosine
    Cosine --> PromptUtils
    PromptUtils --> PromptConfig

    PromptConfig --> LavisInf
    PromptConfig --> LlavaServeCLI
    PromptConfig --> PhiServeInf

    subgraph "Orchestrator & Scripts"
        Main["main.py\n/ main.py"]:::orchestrator
        BLIPRun["BLIP-2 Run Scripts\n(train_caption_xai.sh)"]:::orchestrator
        LlavaRun["LLaVA Scripts\n(pretrain.sh)"]:::orchestrator
        PhiRun["Phi3 Scripts\n(finetune.sh)"]:::orchestrator
    end

    Main --> FrameExt
    Main --> LavisApp
    Main --> LlavaScripts
    Main --> PhiTrain
    Main --> Cosine
    Main -->|"launches scripts"| BLIPRun
    Main --> LlavaRun
    Main --> PhiRun

    subgraph "Evaluation & Results"
        CSV["CSV Writer & Main\n(main.py)"]:::eval
        BranchEval["Branch Eval Scripts\n(evaluate_script.py)"]:::eval
        DBStore["DB Persistence\n(database_mysql.py)"]:::eval
        Notebooks["Jupyter Notebooks"]:::eval
    end

    LavisInf --> CSV
    LlavaServeCLI --> CSV
    PhiServeApp --> CSV
    LavisEval --> CSV
    LavisInf --> BranchEval
    CSV --> DBStore
    CSV --> Notebooks

    click Raw "https://github.com/vladhondru25/exddv/blob/main/ExDDV.csv"
    click FrameExt "https://github.com/vladhondru25/exddv/blob/main/database.py"
    click MySQLDB "https://github.com/vladhondru25/exddv/blob/main/database_mysql.py"
    click LavisInf "https://github.com/vladhondru25/exddv/blob/main/src/LAVIS/inference.py"
    click LavisApp "https://github.com/vladhondru25/exddv/tree/main/src/LAVIS/app/"
    click LavisEvalGen "https://github.com/vladhondru25/exddv/blob/main/src/LAVIS/evaluate_general.py"
    click LavisEval "https://github.com/vladhondru25/exddv/blob/main/src/LAVIS/evaluate_script.py"
    click Llavabuilder "https://github.com/vladhondru25/exddv/blob/main/src/LLaVA/llava/model/builder.py"
    click LlamaEnc "https://github.com/vladhondru25/exddv/tree/main/src/LLaVA/llava/model/multimodal_encoder/"
    click LlavaScripts "https://github.com/vladhondru25/exddv/tree/main/src/LLaVA/scripts/"
    click LlavaServeCLI "https://github.com/vladhondru25/exddv/blob/main/src/LLaVA/llava/serve/cli.py"
    click LlavaController "https://github.com/vladhondru25/exddv/blob/main/src/LLaVA/llava/serve/controller.py"
    click PhiTrain "https://github.com/vladhondru25/exddv/blob/main/src/Phi3-Vision-Finetune/src/training/train.py"
    click PhiMerge "https://github.com/vladhondru25/exddv/blob/main/src/Phi3-Vision-Finetune/src/merge_lora_weights.py"
    click PhiServeInf "https://github.com/vladhondru25/exddv/blob/main/src/Phi3-Vision-Finetune/src/serve/inference.py"
    click PhiServeCLI "https://github.com/vladhondru25/exddv/blob/main/src/Phi3-Vision-Finetune/src/serve/cli.py"
    click PhiServeApp "https://github.com/vladhondru25/exddv/blob/main/src/Phi3-Vision-Finetune/src/serve/app.py"
    click PromptConfig "https://github.com/vladhondru25/exddv/blob/main/src/LAVIS/lavis/projects/blip2/train/pretrain_stage2_xai.yaml"
    click Main "https://github.com/vladhondru25/exddv/blob/main//main.py"
    click BLIPRun "https://github.com/vladhondru25/exddv/blob/main/src/LAVIS/run_scripts/blip2/train/train_caption_xai.sh"
    click LlavaRun "https://github.com/vladhondru25/exddv/blob/main/src/LLaVA/scripts/pretrain.sh"
    click PhiRun "https://github.com/vladhondru25/exddv/blob/main/src/Phi3-Vision-Finetune/scripts/finetune.sh"
    click Notebooks "https://github.com/vladhondru25/exddv/blob/main/blip_incontext.ipynb"
    click Notebooks "https://github.com/vladhondru25/exddv/blob/main/llava_incontext.ipynb"
    click Notebooks "https://github.com/vladhondru25/exddv/blob/main/phi_incontext.ipynb"

    classDef data fill:#f9f,stroke:#333,stroke-width:1px
    classDef utils fill:#ddd,stroke:#333
    classDef storage fill:#bbb,stroke:#333
    classDef lavis fill:#aaf,stroke:#339
    classDef llava fill:#afa,stroke:#393
    classDef phi3 fill:#fea,stroke:#e93
    classDef builder fill:#ccc,stroke:#333
    classDef orchestrator fill:#fcc,stroke:#933
    classDef eval fill:#fcf,stroke:#933
```
