How jamie725/multimodal-object-detection-via-probabilistic-ensembling works
Project Type and Purpose
• This is an ML research codebase built on top of Facebook’s Detectron2 framework.
• Purpose: Train and evaluate object detectors (thermal‐only, early/middle fusion, Bayesian fusion) on FLIR/KAIST datasets, then perform probabilistic ensembling of multiple model outputs.
Main Components
a. Core Framework (Detectron2)
• Configuration subsystem (detectron2/config + YAML files under /configs and projects/*/configs)
• Data pipeline (detectron2/data): dataset registration, mappers, transforms
• Model library (detectron2/modeling + layers): backbones, RPN, ROI heads, mask ops, etc.
• Training engine (detectron2/engine): default Trainer, hooks, launch scripts
• Evaluation modules (detectron2/evaluation + custom FLIR_evaluation.py)
• Export / serialization (detectron2/export, checkpointing)
b. Project Extensions
• Custom evaluation in detectron2/evaluation/FLIR_evaluation.py
• Configuration overrides in probEn.yml and run scripts (run.sh)
c. Supporting Tools & Scripts
• run.sh: top‐level orchestration (sets fusion method, invokes demos)
• Utility scripts: logging, model zoo, TensorBoard events under /detections
• Docker setup for reproducible environments (docker/)
• CI configuration (.circleci/, GitHub issue/pull templates)
Relationships & Data Flow
1. User invokes run.sh or demo/FLIR/demo_train_FLIR.py
2. Config loader reads YAML (fusion method, model path) → builds model & data loader
3. Data loader reads RGB & thermal frames → applies transforms → batches
4. Trainer (Detectron2 engine) executes training loop → writes checkpoints & TensorBoard logs
5. Demo save‐predictions script runs inference on saved models → writes JSON/pickled prediction files
Architectural Patterns & Design Principles
• Plugin/Registry pattern: datasets, models, hooks are registered and instantiated via config
• Layered Modular Design: data → model → engine → evaluation → visualization
• Configuration‐Driven: almost every behavior toggled via YAML or command‐line flags
• Separation of Concerns: Detectron2 core vs. project‐specific extensions vs. demo scripts
Diagramming Recommendations
• Core Detectron2 modules (e.g., blue rectangles)
• Project extensions & demos (e.g., green)
• External artifacts (datasets, checkpoints, logs) (e.g., grey)
• From “Config Files (YAML)” → “Config Loader” → “Model & Data Instantiation”
• From “Dataset (RGB + Thermal)” → “Data Loader” → “Model”
• From “Model” → “CheckpointWriter & TensorBoard”
• From “Inference Scripts” → “Saved Predictions” → “ProbEn Ensembler” → “Fused Outputs” → “Visualizer”
• Related components within containers:
• Core Framework
• Project Extensions
• Deployment & CI Tools