How duongtruongbinh/vivqa-x works
Project Type and Purpose
• This is a research-oriented, end-to-end pipeline designed to construct a Vietnamese Visual Question Answering dataset with natural language explanations (VQA-NLE). Its purpose is not only to build the dataset but also to benchmark different models (heuristic and baseline LSTM-generative) on it.
• The repository includes multiple “subsystems”: dataset management, data processing via an automated pipeline, model benchmarking (training and evaluation), and utilities/scripts for deployment.
Main Components
• Data Layer:
• “data/final” holds the core JSON files (train, validation, test) sourced from COCO with Vietnamese questions, answers, and explanations.
• That there’s an initial download process (using the download_vqax.sh script) to retrieve the dataset.
• Pipeline Module:
• Under “src/pipeline”, the project separates functions into several submodules:
▪ Post-Processing: “post_processing” – this might handle refinement of model outputs or dataset quality.
▪ Selection: “selection” – which further contains evaluators (base, gemma, gpt, llama, etc.) for selecting and ranking generated outputs.
▪ Translation: “translation” – modules that involve data processing for translation (with submodules for different translation engines).
• Models Module:
• Contains two main model directories:
▪ Baseline Model (LSTM-Generative): “src/models/baseline_model”
• Contains configuration (config/config.yaml), training (train.py), evaluation (evaluate.py), and a model definition (vivqax_model.py) along with dataloaders, metrics, and utilities.
▪ Heuristic Model: “src/models/heuristic_model”
• Contains similar structure with configuration, dataloaders, heuristics (heuristic_baseline.py) and a run script (run_heuristic.py).
• Separate branches for these models indicating they both share some standard evaluation metrics (e.g., BLEU, ROUGE, SPICE, etc.) but are triggered by different scripts/entry points.
• Configuration and Scripting:
• “requirements.txt” and configuration YAML files supply runtime parameters including device settings, model hyperparameters, and file paths.
• “scripts/” include shell scripts (pipeline.sh, train.sh, evaluate.sh) to execute the automated workflow steps – from data downloading to training and evaluation.
• How these scripts orchestrate the components, i.e., they serve as entry points that trigger the underlying Python modules in “src”.
Relationships and Interactions
• Data flows from the data layer into the pipeline module where translation, selection, and post-processing prepare the dataset.
• The processed data is then fed into model training (baseline model’s train.py) or evaluated directly by the heuristic model’s run script.
• Evaluators in the “selection” submodule provide feedback (possibly through metric calculations) that might be used for refining model results.
• Configuration files provide the necessary parameters for both the pipeline operations and model execution.
• That training outputs (model weights) and evaluation results are stored and later used for performance assessment.
Architectural Patterns and Design Principles
• Modularity: Each major function (data processing, model training, evaluation, translation, selection) is encapsulated in its own module or directory.
• Config-driven: YAML configuration files drive operational parameters, allowing for flexible experimentation.
• Scripted Orchestration: Shell scripts in the “scripts” folder serve as orchestration wrappers for end-to-end execution.
• Clear separation between rule-based (heuristic) and learning-based (LSTM-generative) approaches.
Elements to Feature
• Components (e.g., “Data Ingestion”, “Translation Module”, “Selection Evaluators”, “Baseline Model Training”, “Heuristic Model Evaluation”, “Post Processing”).
• How the raw dataset (data/final) is processed through the pipeline modules.
• Flow from configuration files to the individual modules.
• Labels and Annotations:
• Evaluation metrics flow (e.g., “Output: BLEU, METEOR, etc.”) emanating from model evaluation blocks.
• Visual Distinctions:
• Optionally, incorporate callouts to indicate the shell scripts that orchestrate these components.