How bsl-traveller/caligen works
Project type and purpose
• This is a research-style machine-learning pipeline (“CaliGen”) for classifier calibration in unseen contexts.
• It’s a command-line-driven, config-based experiment orchestration system with discrete stages (preprocess, train, calibrate, eval), plus Jupyter notebooks for visualization.
File structure
• Entry point: main.py
• Configuration: model/configs/*.py → dataset, model, loss definitions
• Preprocessing: preprocessing/preprocess.py, preprocessing/corruptions.py, data_utils.py
• Model training & evaluation: model/dnn_models.py, model/steps.py, model/model_utils.py
• Calibration logic: calibration/calibrate.py, calibration/cal_methods.py
• Results & artifacts:
• DataSets folder (external raw datasets)
• logs/<workdir> (experiment outputs, checkpoints, calibrated models)
• results/ (aggregated CSVs for different architectures/datasets/losses)
• Visualization: Notebooks/
• Environment setup: environment.yml, requirements.txt
README for additional insights
• CLI modes: preprocess, train, calibrate, eval_performance
• External dataset downloads (CIFAR-10C, Office-Home, DomainNet)
• Outputs go into workdir, then notebooks read logs to produce figures
Main components & their interactions
a. Components:
• CLI Orchestrator (main.py)
• Config Loader (reads model/configs)
• Data Preprocessing Module
• Training Module
• Calibration Module
• Evaluation Module
• Artifact Storage (workdir: raw/preprocessed data, model checkpoints, calibrated weights)
• Metrics & Results Store (results/ folder with CSVs)
• Visualization Layer (Notebooks/)
• External Data Sources (downloaded datasets)
b. Relationships & Data Flow:
1. User invokes CLI → main.py
2. Config Loader parses flags & .py configs → passes parameters to modules
3. Preprocessing Module reads raw DataSets → writes preprocessed data to workdir
4. Training Module consumes preprocessed data → outputs model checkpoints to workdir
5. Calibration Module consumes checkpoints → outputs calibrated weights under workdir/calib_models
6. Evaluation Module uses calibrated weights → writes metrics CSVs under results/
7. Notebooks read metrics & figures from workdir/results for visualization
8. All modules log to file system; no direct network calls except initial dataset download
c. Architectural patterns & design principles:
• Modular pipeline – each stage is a separate component, invoked by a single orchestrator
• Config-driven – experiment behavior is determined by Python config files
• Separation of concerns – preprocessing, training, calibration, evaluation, and visualization are distinct
• Reproducibility – environment.yml and workdir snapshot per run
d. Key technologies & libraries:
• PyTorch or TensorFlow (implied by DNN modules)
• Hydra-style or argparse config loading
• pandas/numpy for CSV results
• Jupyter for notebooks
• CLI entry via python -m main
Detail
• Every major directory as a component if seen in the file tree
• How model/configs feeds into each stage
• Which scripts or packages live in each component