# jamie725/multimodal-object-detection-via-probabilistic-ensembling architecture

> Architecture diagram of the GitHub repository jamie725/multimodal-object-detection-via-probabilistic-ensembling, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling
- Repository: https://github.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling

## Overview

##### Project Type and Purpose
- This is an ML research codebase built on top of Facebook’s Detectron2 framework.
- Purpose: Train and evaluate object detectors (thermal‐only, early/middle fusion, Bayesian fusion) on FLIR/KAIST datasets, then perform probabilistic ensembling of multiple model outputs.

##### Main Components
a. Core Framework (Detectron2)
- Configuration subsystem (detectron2/config + YAML files under `/configs` and `projects/*/configs`)
- Data pipeline (detectron2/data): dataset registration, mappers, transforms
- Model library (detectron2/modeling + layers): backbones, RPN, ROI heads, mask ops, etc.
- Training engine (detectron2/engine): default Trainer, hooks, launch scripts
- Evaluation modules (detectron2/evaluation + custom `FLIR_evaluation.py`)
- Export / serialization (detectron2/export, checkpointing)

b. Project Extensions
- Custom evaluation in `detectron2/evaluation/FLIR_evaluation.py`
- Configuration overrides in `probEn.yml` and run scripts (`run.sh`)

c. Supporting Tools & Scripts
- `run.sh`: top‐level orchestration (sets fusion method, invokes demos)
- Utility scripts: logging, model zoo, TensorBoard events under `/detections`
- Docker setup for reproducible environments (`docker/`)
- CI configuration (`.circleci/`, GitHub issue/pull templates)

##### Relationships & Data Flow
1. User invokes `run.sh` or `demo/FLIR/demo_train_FLIR.py`
2. Config loader reads YAML (fusion method, model path) → builds model & data loader
3. Data loader reads RGB & thermal frames → applies transforms → batches
4. Trainer (Detectron2 engine) executes training loop → writes checkpoints & TensorBoard logs
5. Demo save‐predictions script runs inference on saved models → writes JSON/pickled prediction files

##### Architectural Patterns & Design Principles
- Plugin/Registry pattern: datasets, models, hooks are registered and instantiated via config
- Layered Modular Design: data → model → engine → evaluation → visualization
- Configuration‐Driven: almost every behavior toggled via YAML or command‐line flags
- Separation of Concerns: Detectron2 core vs. project‐specific extensions vs. demo scripts

##### Diagramming Recommendations
- Core Detectron2 modules (e.g., blue rectangles)
- Project extensions & demos (e.g., green)
- External artifacts (datasets, checkpoints, logs) (e.g., grey)
- From “Config Files (YAML)” → “Config Loader” → “Model & Data Instantiation”
- From “Dataset (RGB + Thermal)” → “Data Loader” → “Model”
- From “Model” → “CheckpointWriter & TensorBoard”
- From “Inference Scripts” → “Saved Predictions” → “ProbEn Ensembler” → “Fused Outputs” → “Visualizer”
- Related components within containers:
- Core Framework
- Project Extensions
- Deployment & CI Tools

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Core Framework
    subgraph "Core Detectron2 Framework"
        direction TB
        ConfigLoader["Config Loader"]:::core
        DataPipeline["Data Pipeline"]:::core
        ModelLib["Model Library"]:::core
        TrainingEngine["Training Engine"]:::core
        EvalModules["Evaluation Modules"]:::core
        Export["Export/Serialization"]:::core
    end

    %% Project Extensions
    subgraph "Project Extensions"
        direction TB
        FLIRDemo["FLIR Demo Scripts"]:::project
        KAISTDemo["KAIST Demo Scripts"]:::project
        ProbEn["ProbEn Ensembler"]:::project
        FusionConfig["Fusion Config Override"]:::project
        RunScript["Top-Level Orchestration (run.sh)"]:::project
    end

    %% External Artifacts & Tools
    subgraph "Supporting Tools & External Artifacts"
        direction TB
        Dataset["Dataset (RGB + Thermal)"]:::external
        ConfigFiles["Config Files (YAML)"]:::external
        SavedPred["Saved Predictions"]:::external
        FusedOutputs["Fused Detections"]:::external
        Checkpoint["Checkpoints & TensorBoard Logs"]:::external
        Visualizer["Visualizer Scripts"]:::external
        TBEvents["TensorBoard Events"]:::external
        Docker["Docker Setup"]:::external
        CI["CI Configuration"]:::external
    end

    %% Data & Control Flow
    ConfigFiles -->|"loads config"| ConfigLoader
    ConfigLoader -->|"instantiates"| DataPipeline
    Dataset -->|"images + metadata"| DataPipeline
    DataPipeline -->|"batches"| ModelLib
    ConfigLoader -->|"builds model"| ModelLib
    ModelLib -->|"training loop"| TrainingEngine
    TrainingEngine -->|"writes pth"| Checkpoint
    TrainingEngine -->|"logs metrics"| TBEvents
    TrainingEngine -->|"hooks eval"| EvalModules
    TrainingEngine -->|"exports model"| Export

    %% Demo and Ensembling Flow
    RunScript -->|"invokes"| FLIRDemo
    RunScript -->|"invokes"| KAISTDemo
    FLIRDemo -->|"inference"| SavedPred
    KAISTDemo -->|"inference"| SavedPred
    SavedPred -->|"reads predictions"| ProbEn
    FusionConfig -->|"configures"| ProbEn
    ProbEn -->|"outputs"| FusedOutputs
    FusedOutputs -->|"visualize"| Visualizer

    %% Supporting Tools Links
    RunScript --- Docker
    RunScript --- CI

    %% Click Events
    click ConfigLoader "https://github.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling/tree/main/detectron2/config/"
    click DataPipeline "https://github.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling/tree/main/detectron2/data/"
    click ModelLib "https://github.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling/tree/main/detectron2/modeling/"
    click TrainingEngine "https://github.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling/tree/main/detectron2/engine/"
    click EvalModules "https://github.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling/tree/main/detectron2/evaluation/"
    click Export "https://github.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling/tree/main/detectron2/export/"
    click FLIRDemo "https://github.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling/tree/main/demo/FLIR/"
    click KAISTDemo "https://github.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling/tree/main/demo/KAIST/"
    click ProbEn "https://github.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling/blob/main/demo/FLIR/demo_probEn.py"
    click FusionConfig "https://github.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling/blob/main/probEn.yml"
    click RunScript "https://github.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling/blob/main/run.sh"
    click TBEvents "https://github.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling/tree/main/detections/"
    click Docker "https://github.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling/tree/main/docker/"
    click CI "https://github.com/jamie725/multimodal-object-detection-via-probabilistic-ensembling/tree/main/.circleci/"

    %% Styles
    classDef core fill:#BBDEFB,stroke:#0D47A1,stroke-width:2px
    classDef project fill:#C8E6C9,stroke:#1B5E20,stroke-width:2px
    classDef external fill:#EEEEEE,stroke:#424242,stroke-width:2px
```
