# mavceleb/mavceleb_baseline architecture

> Architecture diagram of the GitHub repository mavceleb/mavceleb_baseline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mavceleb/mavceleb_baseline
- Repository: https://github.com/mavceleb/mavceleb_baseline

## Overview

- Data & Input Sources:
- Dataset: “dataset” directory containing raw data (zips, raw images, and audio).
- Pre-extracted Features: “preExtracted_vggFace_utteranceLevel_Features” directory holding v1 and v2 CSV/TXT files.
- Split Files: “face_voice_association_splits” directory with train/test split text files.
- Feature Extraction Modules:
- Face Feature Extraction: “vggFaceFeat.py” which uses VGGFace to produce 4096-D embeddings.
- Voice Feature Extraction: “uttLevelVoiceFeat.py” (referencing the VGG-Speaker-Recognition based method) that produces 512-D embeddings.
- Model Directories:
- “v1_models” and “v2_models”: These hold trained FOP models for different language scenarios (English, Urdu, Hindi).
- Core Execution Scripts:
- “main.py”: Likely the entry point to the overall system.
- “retrieval_model.py”: Possibly handling retrieval or matching operations based on extracted features.
- “test.py”: For running evaluation experiments with the pre-extracted features.
- “computeScore.py”: For calculating and generating submission scores based on evaluation outputs.

##### Data Flows & Interactions
- That these features are either stored in preExtracted directories (as intermediate outputs) or fed directly into the model.
- “retrieval_model.py” component for processing or matching face-voice pairs.

##### Architectural Patterns/Design Principles
- This system follows a modular pipeline approach:
- Separation of feature extraction,
- Model inference (using FOP models for association),
- And evaluation (scoring mechanism).
- The design encapsulates the cross-modal verification task where each module has a clear responsibility.
- Notice a dual set of models (v1 and v2) to evaluate different configurations (languages and experimental setups).

##### Detailed Components
a. Input Stage:
– Raw data from “dataset” and split information from “face_voice_association_splits”.
b. Feature Extraction Stage:
– Face Feature Extraction (vggFaceFeat.py)
– Voice Feature Extraction (uttLevelVoiceFeat.py)
c. Model Stage:
– FOP Models (v1_models and v2_models directories indicate two versions,
with checkpoints for different language training)
d. Inference/Retrieval Stage:
– “retrieval_model.py” that likely utilizes the FOP method to compute and compare embeddings.
e. Evaluation Stage:
– “test.py” orchestrates the evaluation on test splits.
– “computeScore.py” computes the final L2 scores for heard versus unheard languages.
f. Auxiliary:
– “main.py”: which may serve as an overall driver for the system.

## Diagram (Mermaid)

```mermaid
flowchart TD
    main["Main Execution Driver (main.py)"]:::driver
    subgraph "Input Stage"
        dataset["Dataset (Raw Data)"]:::datasource
        splits["Split Files (Train/Test)"]:::datasource
    end
    subgraph "Feature Extraction Stage"
        vgg["Face Feature Extraction (vggFaceFeat.py)"]:::process
        utt["Voice Feature Extraction (uttLevelVoiceFeat.py)"]:::process
    end
    preEx["Pre-extracted Features Storage"]:::datasource
    subgraph "FOP Models Stage"
        fopV1["FOP Model (v1_models)"]:::model
        fopV2["FOP Model (v2_models)"]:::model
    end
    retrieval["Retrieval & Matching (retrieval_model.py)"]:::process
    subgraph "Evaluation Stage"
        test["Evaluation Script (test.py)"]:::evaluation
        score["Score Computation (computeScore.py)"]:::evaluation
    end

    %% Data flow connections
    main --> dataset
    main --> splits
    dataset -->|"RawData"| vgg
    dataset -->|"RawData"| utt
    splits -->|"SplitInfo"| vgg
    splits -->|"SplitInfo"| utt
    vgg -->|"ExtractFaceFeatures"| preEx
    utt -->|"ExtractVoiceFeatures"| preEx
    preEx -->|"FeatureInput"| fopV1
    preEx -->|"FeatureInput"| fopV2
    fopV1 -->|"FOPAssociation"| retrieval
    fopV2 -->|"FOPAssociation"| retrieval
    retrieval -->|"MatchingResults"| test
    test -->|"EvalData"| score

    %% Click Events
    click dataset "https://github.com/mavceleb/mavceleb_baseline/tree/main/dataset/"
    click splits "https://github.com/mavceleb/mavceleb_baseline/tree/main/face_voice_association_splits/"
    click vgg "https://github.com/mavceleb/mavceleb_baseline/blob/main/vggFaceFeat.py"
    click utt "https://github.com/mavceleb/mavceleb_baseline/blob/main/uttLevelVoiceFeat.py"
    click preEx "https://github.com/mavceleb/mavceleb_baseline/tree/main/preExtracted_vggFace_utteranceLevel_Features/"
    click fopV1 "https://github.com/mavceleb/mavceleb_baseline/tree/main/v1_models/"
    click fopV2 "https://github.com/mavceleb/mavceleb_baseline/tree/main/v2_models/"
    click retrieval "https://github.com/mavceleb/mavceleb_baseline/blob/main/retrieval_model.py"
    click test "https://github.com/mavceleb/mavceleb_baseline/blob/main/test.py"
    click score "https://github.com/mavceleb/mavceleb_baseline/blob/main/computeScore.py"
    click main "https://github.com/mavceleb/mavceleb_baseline/blob/main/main.py"

    %% Styles
    classDef datasource fill:#A3BE8C,stroke:#2E3440,stroke-width:2px;
    classDef process fill:#88C0D0,stroke:#2E3440,stroke-width:2px;
    classDef model fill:#BF616A,stroke:#2E3440,stroke-width:2px;
    classDef evaluation fill:#EBCB8B,stroke:#2E3440,stroke-width:2px;
    classDef driver fill:#D08770,stroke:#2E3440,stroke-width:2px;
```
