How zijialewislu/cvpr2024-fact works
Project's Purpose and Type
• The README clearly indicates that it’s an action segmentation framework built for processing video datasets (Breakfast, GTEA, EgoProceL, Epic-Kitchens).
• The main functionalities include training (train.py), evaluation (eval.py), handling various configurations (configs directory), and model definition (models directory).
• It’s a full-stack ML project with well-separated components for configuration, model, dataset handling, training, and evaluation.
File Structure and Map It to Architectural Components
• Top-level files like train.py, eval.py, and home.py set the entry points for training and evaluation.
• The "configs" folder contains YAML configuration files (e.g., breakfast.yaml, gtea.yaml, etc.) that dictate experiment parameters – consider this as the Config Manager.
• The "models" folder defines the learning models, with files that establish core building blocks (blocks.py, blocks_SepVerbNoun.py) and loss computation (loss.py). This is your Model Component.
• The "utils" folder includes scripts for dataset preparation (dataset.py), experiment evaluation (evaluate.py), training utilities (train_tools.py), and other helper functions (utils.py) – these form the Utility Engine supporting data operations, training flow, and analytics.
• The README also references external data preparation (download/extraction scripts in extract_epic_kitchens.py) and integration with wandb for metrics visualization.
Main Components and Their Interactions
• Data Input & Preprocessing:
• The system will ingest datasets (Breakfast, GTEA, etc.) that are to be stored in a designated data folder (created separately).
• Use the utils/dataset.py module along with extraction scripts (e.g., extract_epic_kitchens.py) to load and process raw video features.
• Configuration Management:
• The configs folder (with YAML files and helper utilities in configs/utils.py) acts as the central manager for all experiment settings.
• Model Definition & Architecture:
• The models folder defines various architectural components, including basic model structures and specialized blocks for cross-attention modeling.
• The loss module within models ensures the network is optimized correctly.
• Training Pipeline:
• The main training logic resides in train.py, which leverages configurations from the configs folder and model definitions from the models folder.
• Supplementary tools in utils/train_tools.py help drive the training process.
• Evaluation and Logging:
• Evaluation is handled by eval.py and supporting utilities found in utils/evaluate.py.
• The integration with wandb is used for logging metrics, visualizing losses, and tracking experiments.
• External Dependencies and Data Flow:
• Pre-trained models can be loaded from ckpts, and data preparation steps are partly defined via Git commands and scripts in README.