How mavceleb/mavceleb_baseline works
• Data & Input Sources:
• Dataset: “dataset” directory containing raw data (zips, raw images, and audio).
• Pre-extracted Features: “preExtracted_vggFace_utteranceLevel_Features” directory holding v1 and v2 CSV/TXT files.
• Split Files: “face_voice_association_splits” directory with train/test split text files.
• Feature Extraction Modules:
• Face Feature Extraction: “vggFaceFeat.py” which uses VGGFace to produce 4096-D embeddings.
• Voice Feature Extraction: “uttLevelVoiceFeat.py” (referencing the VGG-Speaker-Recognition based method) that produces 512-D embeddings.
• Model Directories:
• “v1_models” and “v2_models”: These hold trained FOP models for different language scenarios (English, Urdu, Hindi).
• Core Execution Scripts:
• “main.py”: Likely the entry point to the overall system.
• “retrieval_model.py”: Possibly handling retrieval or matching operations based on extracted features.
• “test.py”: For running evaluation experiments with the pre-extracted features.
• “computeScore.py”: For calculating and generating submission scores based on evaluation outputs.
Data Flows & Interactions
• That these features are either stored in preExtracted directories (as intermediate outputs) or fed directly into the model.
• “retrieval_model.py” component for processing or matching face-voice pairs.
Architectural Patterns/Design Principles
• This system follows a modular pipeline approach:
• Separation of feature extraction,
• Model inference (using FOP models for association),
• And evaluation (scoring mechanism).
• The design encapsulates the cross-modal verification task where each module has a clear responsibility.
• Notice a dual set of models (v1 and v2) to evaluate different configurations (languages and experimental setups).
Detailed Components
a. Input Stage:
– Raw data from “dataset” and split information from “face_voice_association_splits”.
b. Feature Extraction Stage:
– Face Feature Extraction (vggFaceFeat.py)
– Voice Feature Extraction (uttLevelVoiceFeat.py)
c. Model Stage:
– FOP Models (v1_models and v2_models directories indicate two versions,
with checkpoints for different language training)
d. Inference/Retrieval Stage:
– “retrieval_model.py” that likely utilizes the FOP method to compute and compare embeddings.
e. Evaluation Stage:
– “test.py” orchestrates the evaluation on test splits.
– “computeScore.py” computes the final L2 scores for heard versus unheard languages.
f. Auxiliary:
– “main.py”: which may serve as an overall driver for the system.