How brendanartley/byu-competition works
Project Type and Purpose
• This is a full-stack machine‐learning training and inference pipeline tailored to a 3D U-Net solution for a Kaggle competition (Locating Bacterial Flagellar Motors).
• It orchestrates data ingestion, preprocessing, augmentation, model definition (3D U-Net and ResNet backbones), training, logging, checkpointing, and inference/thresholding.
File Structure Analysis
• Top-level:
• run.sh, train.py – orchestration scripts for training/end-to-end runs
• requirements.txt – Python dependencies
• README.md – high-level instructions and solution summary
• imgs/ – architecture illustration used in docs
• src/ directory (core codebase):
• src/pre/ – preprocessing entry point (src/pre/run.py)
• src/data/ – data loaders, utilities, raw↔processed paths
• src/augs/ – 3D data augmentation modules (mixup, cutmix, custom 3D)
• src/models/ – model definitions: base classes, 3D ResNet layers, U-Net layers
• src/configs/ – Hydra/Config-style experiment configuration
• src/losses/ – custom and auxiliary loss functions
• src/modules/ – higher-level training and metric loops
• src/logging/ – wandb or custom logging utilities
• src/utils/ – general utilities (cfg, torch helpers)
• Data directory (external, described in README): ./data/{raw,processed,checkpoints,model_zoo}
a. Data Ingestion & Preprocessing
– Raw competition tomograms (./data/raw → src/pre/run.py → processed .npy files)
b. Data Augmentation Pipeline
– src/augs/{mixup,cutmix,aug3d} applied inside data loader
c. Model Definition
– 3D U-Net (src/models/unet3d.py) built from layers in src/models/layers
– Optional auxiliary ResNet3D backbone (configs control r3d18 vs r3d200)
d. Loss Computation
– Primary segmentation loss + auxiliary losses (src/losses/base.py)
e. Training Loop & Logging
– src/modules/train.py orchestrates forward/backward, metrics (src/modules/metric.py), wandb logging (src/logging/utils.py)
f. Checkpointing & Model Zoo
– Checkpoints written to ./data/checkpoints, pretrained backbones loaded from model_zoo
g. Inference & Post-processing
– Ranking tomograms by max prediction, quantile thresholding for final binary decision
Relationships & Interactions
• Data flows: Raw files → preprocessing → DataLoader → augmentations → model → loss → gradients → update → checkpoint
• Configuration flows: configs load hyperparameters → drive model instantiation, optimizer, training parameters
• Logging: metrics & artifacts streamed to WandB via logging utils
• External services: GPU (CUDA), conda environment, WandB cloud
Architectural Patterns & Design Principles
• Modular layering: separate folders for augmentations, models, losses, modules
• Config-driven experiments: reuse of base and override configs for different backbones
• Clear separation of concerns: preprocessing, training, inference components are decoupled
• Utility code centralized under src/utils and src/logging
• Clear labels (e.g., “src/pre/run.py: Preprocessing”)
• Artifacts: config files, checkpoints, model_zoo, WandB