How ybhbingo/mmfi_dataset works
1. Repository purpose
- This is a dataset toolbox for MM-Fi, not a full training framework.
- Its main job is to turn a disk-backed multimodal dataset plus a YAML config into PyTorch-ready datasets and dataloaders, with evaluation support.
2. Top-level entry points
- example.py is the usage example and integration template.
- config.yaml is the main control plane for dataset selection, split policy, modality choice, and dataloader settings.
- README.md defines the expected dataset layout and the supported configuration semantics.
3. Core library boundary
- The reusable logic lives under mmfi_lib/.
- mmfi_lib/mmfi.py is the dataset-facing API surface; the README shows it exposing make_dataset and make_dataloader.
- mmfi_lib/evaluate.py is the evaluation subsystem, likely responsible for metric computation on model outputs versus MM-Fi annotations.
4. Data source and storage model
- The system assumes a hierarchical on-disk dataset rooted at ${DATASET_ROOT} with paths organized by environment, subject, and action:
- environment group E01–E04
- subject S01–S10
- action A01–A27
- modality subfolders such as rgb, mmwave, wifi-csi, depth, lidar, infra1, infra2
- This filesystem structure is the primary data boundary: the library reads local files directly rather than calling a service.
5. Configuration-driven behavior
- modality controls whether a single modality or a fused multimodal sample is loaded.
- data_unit switches between per-frame and per-sequence sampling, which changes the granularity of each dataset item.
- protocol filters which action classes are in scope: daily, rehabilitation, or all actions.
- split selects a predefined split or a custom manual split for train/validation or train/test partitioning.
- train_loader and validation_loader expose standard PyTorch dataloader knobs without hardcoding them into the library.
6. Dataset construction flow
- The typical flow is:
- load YAML config
- call make_dataset(dataset_root, config)
- receive train and validation dataset objects
- This means the library is responsible for:
- interpreting the split definition
- filtering by protocol and modality
- mapping disk files to sample records
- producing dataset objects that can be wrapped by PyTorch loaders
7. Batching and runtime integration
- make_dataloader(...) wraps the dataset objects into PyTorch dataloaders.
- The example seeds a Torch RNG and passes the same generator into both loaders, indicating reproducible shuffling / sampling is part of the intended runtime behavior.
- The repo therefore sits at the boundary between raw multimodal files and model training code.
8. Annotation and label handling
- MM-Fi includes pose and action annotations, but the toolbox is oriented around dataset access rather than annotation generation.
- The README notes that some RGB modalities are privacy-restricted and replaced with extracted keypoints, which affects how the loader should treat modality availability and feature format.
- The action label taxonomy is fixed at 27 categories and is used to define protocol filtering.
9. Evaluation subsystem
- mmfi_lib/evaluate.py is a separate concern from loading.
- Architecturally, this suggests evaluation is decoupled from dataset construction so that downstream projects can reuse the metrics without depending on the full data-loading path.
10. External dependencies that shape the design
- The toolbox is built around Python, PyTorch, YAML, NumPy/SciPy, and OpenCV.
- PyTorch drives dataset/dataloader abstractions.
- YAML drives configuration.
- NumPy/SciPy/OpenCV imply preprocessing and file-format handling inside the library, likely for numerical signal modalities and image-like inputs.
11. What is intentionally not in scope
- There is no evidence of a training loop, model definition, or experiment manager in this repository.
- The repo is focused on data access, split management, and evaluation hooks, leaving modeling and optimization to downstream projects.
12. How to diagram it
- Draw three main blocks:
- disk dataset tree plus config file as inputs
- mmfi_lib/mmfi.py as the loader/factory layer producing datasets and dataloaders
- mmfi_lib/evaluate.py as a separate metrics/evaluation layer
- Place example.py as a consumer of the library, showing the intended integration path into external model code.
Main entry points: mmfi.py MM-Fi API · example.py Example · config.yaml Config · evaluate.py Evaluate
Components 17
Inputs
Repository entry points and on-disk dataset shape
- Dataset root · Hierarchical MM-Fi files under environment, subject, action, and modality folders
- Layout spec · E01-E04, S01-S10, A01-A27, and modality subfolders
README.md - Config · Selects modality, data unit, protocol, split, and loader settings
config.yaml - README · Documents dataset organization and configuration semantics
README.md
Core lib
Dataset factory and evaluation logic
- MM-Fi API · Public surface for make_dataset and make_dataloader
mmfi_lib/mmfi.py - Evaluate · Evaluation logic for predictions against MM-Fi annotations
mmfi_lib/evaluate.py - Protocol · Limits scope to daily, rehabilitation, or all actions
- Split policy · Resolves predefined or manual train/validation or train/test splits
- Modality load · Loads single or fused multimodal samples from disk
- Labels · Maps fixed action annotations and pose-related metadata into samples
- Train set · Training dataset object returned by the factory
- Val set · Validation or test dataset object returned by the factory
Runtime use
Example integration and downstream consumption
- Example · Shows dataset construction, loader creation, and seeding
example.py - Train loader · Batches training samples with configured loader knobs
- Val loader · Batches validation samples with configured loader knobs
- Model code · Downstream training or inference code that consumes loaders
- Metrics · Computed scores from evaluator over predictions and annotations
Connections 24
- README → Layout spec: documents · Explains the expected directory hierarchy
- README → Config: documents · Explains supported configuration semantics
- Example → Config: loads · Reads YAML settings before building datasets
- Example → MM-Fi API: calls · Invokes dataset and dataloader factory functions
- Dataset root → MM-Fi API: reads · MM-Fi files are loaded directly from disk
- Config → MM-Fi API: controls · Drives modality, protocol, split, and loader behavior
- MM-Fi API → Protocol: applies · Restricts action categories by protocol
- MM-Fi API → Split policy: applies · Chooses predefined or manual partitions
- MM-Fi API → Modality load: builds · Constructs sample records from one or more modalities
- MM-Fi API → Labels: uses · Attaches labels and annotation-derived targets
- Protocol → Train set: feeds · Filtered classes flow into the training dataset
- Split policy → Train set: produces · Creates the training partition
- Split policy → Val set: produces · Creates the validation or test partition
- Modality load → Train set: materializes · Backs training samples with loaded features
- Modality load → Val set: materializes · Backs validation samples with loaded features
- Train set → Train loader: wraps · Training dataset is wrapped by a PyTorch dataloader
- Val set → Val loader: wraps · Validation dataset is wrapped by a PyTorch dataloader
- Example → Train loader: uses · Example shows runtime integration of the training loader
- Example → Val loader: uses · Example shows runtime integration of the validation loader
- Train loader → Model code: feeds · Batches are consumed by external model code
- Val loader → Model code: feeds · Validation batches are consumed by external model code
- Example → Evaluate: imports · Evaluation is used as a separate subsystem
- Model code → Evaluate: evaluates · Predictions are scored against MM-Fi annotations
- Evaluate → Metrics: emits · Produces evaluation metrics and summaries