# sairam13001/c2fdrone architecture

> Architecture diagram of the GitHub repository sairam13001/c2fdrone, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sairam13001/c2fdrone
- Repository: https://github.com/sairam13001/c2fdrone

## Overview

##### Project Type and Purpose
- This is a Python-based deep-learning training and evaluation pipeline for “Drone-to-Drone-Detection” (DDoD).
- It orchestrates data loading, model definition (Deformable DETR + Swin Transformer variants), custom CUDA/CPU ops, training loops, distributed launch scripts, and logging/checkpointing.

##### High-Level Architectural Breakdown
- **Core Components**
a. Data Ingestion Layer
- `datasets/` + `dataset_dets.py` + specialized parsers & samplers
- `data_prefetcher.py` for asynchronous batching
b. Model Definition Layer
- `models/` folder including:
– `backbone.py`, `swin_transformer.py`, `deformable_detr.py`, `matcher.py`, `segmentation.py`
– `models/ops/` – custom CUDA/CPU kernels for MultiScaleDeformableAttention
c. Engine & Orchestration
- `engine.py` (or `engine_single.py`) with `train_one_epoch`, `evaluate` loops
- `main.py` entrypoint wiring config ➔ datasets ➔ model ➔ engine
d. Configuration & Launch
- `configs/` shell scripts (`*_train.sh`, `*_eval.sh`) define hyper-parameters, EXP_DIR, GPUs
- `tools/launch.py` + `tools/run_dist_*.sh` for single-/multi-node distributed launch
e. Utilities & Logging
- Logging to disk (EXP_DIR) and checkpoint saving in `run.sh` or within engine

##### Relationships & Data Flow
[ Raw Video / COCO Files ]
↓ parsed by `coco_video_parser.py` / `vid_single.py`
↓ transforms & samplers
↓ `data_prefetcher` feeds batches
↓ Model Forward Pass (`models/…`)
↓ Loss & Matching (`matcher.py`)
↓ Backprop ↔ Custom Operators (`models/ops/`)
↓ Optimizer step
↓ Checkpoint & Logs
- Control Flow:
[ run_dist_launch.sh ] → spawns N processes running `main.py` → each process sets up DDP, executes `engine.py`

##### Key Architectural Patterns & Principles
- **Layered Design**: clear separation between data, model, ops, engine, tools.
- **Modularity**: interchangeable backbones (Swin variants), dataset flags (temporal fusion).
- **Custom Extension Points**: C++/CUDA ops under `models/ops` compiled via `setup.py` / `make.sh`.
- **Distributed Training Pattern**: each GPU process runs a mirror of the same script under PyTorch DDP.
- **Config-Driven**: shell scripts + environment variables define hyperparams, logging paths.

##### Technologies & Frameworks
- Python 3.7+, PyTorch (with DDP)
- CUDA/C++ for custom kernels
- Docker & nvidia-docker for environment
- Slurm / custom scripts for cluster launch
- Standard libs: argparse, logging, shell scripting

##### Final Recommendations
1) Macro-view: Data → Training Service → Checkpoint & Logs → External (Docker/Cluster)
2) Meso-view: inside Training Service show Model, Engine, Distributed Launcher, Ops
- Finally, export in SVG or PDF for clarity and easy updates.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Ingestion Layer"
        direction TB
        RawData["Raw Video / COCO Files"]:::external
        DatasetDets["dataset_dets.py"]:::data
        Datasets["datasets/"]:::data
        DataPrefetcher["data_prefetcher.py"]:::data
    end

    %% Model & Ops Layer
    subgraph "Model & Ops Layer"
        direction TB
        subgraph "Model Definition" 
            direction TB
            Backbone["backbone.py"]:::model
            Swin["swin_transformer.py"]:::model
            DeformableDETR["deformable_detr.py"]:::model
            DeformableTransformer["deformable_transformer.py"]:::model
            Matcher["matcher.py"]:::model
            Segmentation["segmentation.py"]:::model
            PosEncoding["position_encoding.py"]:::model
            Pseudocode["pseudocode_defDETR.py"]:::model
        end
        subgraph "Custom Ops Module"
            direction TB
            OpsSrc["models/ops/src/"]:::model
            OpsFunc["ms_deform_attn_func.py"]:::model
            OpsModule["ms_deform_attn.py"]:::model
            OpsSetup["setup.py"]:::model
            OpsMake["make.sh"]:::model
            OpsBuild["build/"]:::model
        end
    end

    %% Engine & Orchestration
    subgraph "Training Service"
        direction TB
        LaunchScript["run_dist_launch.sh"]:::control
        LaunchPy["launch.py"]:::control
        Main["main.py"]:::control
        Engine["engine.py"]:::control
    end

    %% Configs
    subgraph "Configuration Scripts"
        direction TB
        SwinSTrain["swins_train.sh"]:::tools
        SwinSEval["swins_eval.sh"]:::tools
        SwinBTrain["swinb_train.sh"]:::tools
        SwinBEval["swinb_eval.sh"]:::tools
    end

    %% Utilities & Logging
    subgraph "Utilities & Logging"
        direction TB
        UtilFolder["util/"]:::tools
        Logging["Logging & Checkpoint Manager"]:::tools
    end

    %% Miscellaneous
    subgraph "Miscellaneous"
        direction TB
        Benchmark["benchmark.py"]:::tools
        RunSh["run.sh"]:::tools
        Requirements["requirements.txt"]:::tools
        DistEggs["dist/"]:::tools
    end

    %% External
    subgraph "External Services"
        direction TB
        Docker["Docker / nvidia-docker"]:::external
        Slurm["Slurm Cluster"]:::external
    end

    %% Connections - Data Flow
    RawData -->|parsed by| DatasetDets
    DatasetDets --> Datasets
    Datasets --> DataPrefetcher
    DataPrefetcher -->|Tensor batches| Backbone
    Backbone --> Swin
    Swin --> DeformableDETR
    DeformableDETR --> Matcher
    Matcher -->|loss & match| Engine
    DeformableDETR --> OpsModule
    OpsModule --> OpsSrc
    OpsSrc --> OpsBuild
    OpsModule -->|custom kernels| OpsFunc
    OpsFunc -->|functions| DeformableDETR

    %% Control Flow
    LaunchScript -->|spawns| Main
    LaunchPy --> Main
    Main --> Engine
    Engine --> Backbone
    Engine -->|checkpoint & logs| Logging

    %% Config injection
    SwinSTrain --> Main
    SwinSEval --> Main
    SwinBTrain --> Main
    SwinBEval --> Main

    %% Utils & Misc
    UtilFolder --> Engine
    Benchmark -->|benchmark| Engine
    RunSh --> Engine
    Requirements -->|deps| Main
    DistEggs --> OpsModule

    %% External services
    Docker --> Engine
    Slurm --> LaunchScript

    %% Click Events
    click DatasetDets "https://github.com/sairam13001/c2fdrone/blob/main/dataset_dets.py"
    click Datasets "https://github.com/sairam13001/c2fdrone/tree/main/datasets/"
    click DataPrefetcher "https://github.com/sairam13001/c2fdrone/blob/main/datasets/data_prefetcher.py"
    click Backbone "https://github.com/sairam13001/c2fdrone/blob/main/models/backbone.py"
    click Swin "https://github.com/sairam13001/c2fdrone/blob/main/models/swin_transformer.py"
    click DeformableDETR "https://github.com/sairam13001/c2fdrone/blob/main/models/deformable_detr.py"
    click DeformableTransformer "https://github.com/sairam13001/c2fdrone/blob/main/models/deformable_transformer.py"
    click Matcher "https://github.com/sairam13001/c2fdrone/blob/main/models/matcher.py"
    click Segmentation "https://github.com/sairam13001/c2fdrone/blob/main/models/segmentation.py"
    click PosEncoding "https://github.com/sairam13001/c2fdrone/blob/main/models/position_encoding.py"
    click Pseudocode "https://github.com/sairam13001/c2fdrone/blob/main/models/pseudocode_defDETR.py"
    click OpsSrc "https://github.com/sairam13001/c2fdrone/tree/main/models/ops/src/"
    click OpsFunc "https://github.com/sairam13001/c2fdrone/blob/main/models/ops/functions/ms_deform_attn_func.py"
    click OpsModule "https://github.com/sairam13001/c2fdrone/blob/main/models/ops/modules/ms_deform_attn.py"
    click OpsSetup "https://github.com/sairam13001/c2fdrone/blob/main/models/ops/setup.py"
    click OpsMake "https://github.com/sairam13001/c2fdrone/blob/main/models/ops/make.sh"
    click OpsBuild "https://github.com/sairam13001/c2fdrone/tree/main/models/ops/build/"
    click Engine "https://github.com/sairam13001/c2fdrone/blob/main/engine.py"
    click Main "https://github.com/sairam13001/c2fdrone/blob/main/main.py"
    click SwinSTrain "https://github.com/sairam13001/c2fdrone/blob/main/configs/swins_train.sh"
    click SwinSEval "https://github.com/sairam13001/c2fdrone/blob/main/configs/swins_eval.sh"
    click SwinBTrain "https://github.com/sairam13001/c2fdrone/blob/main/configs/swinb_train.sh"
    click SwinBEval "https://github.com/sairam13001/c2fdrone/blob/main/configs/swinb_eval.sh"
    click LaunchPy "https://github.com/sairam13001/c2fdrone/blob/main/tools/launch.py"
    click LaunchScript "https://github.com/sairam13001/c2fdrone/blob/main/tools/run_dist_launch.sh"
    click UtilFolder "https://github.com/sairam13001/c2fdrone/tree/main/util/"
    click Benchmark "https://github.com/sairam13001/c2fdrone/blob/main/benchmark.py"
    click RunSh "https://github.com/sairam13001/c2fdrone/blob/main/run.sh"
    click Requirements "https://github.com/sairam13001/c2fdrone/blob/main/requirements.txt"
    click DistEggs "https://github.com/sairam13001/c2fdrone/tree/main/dist/"

    %% Styles
    classDef data fill:#cce5ff,stroke:#004085,color:#004085
    classDef model fill:#d4edda,stroke:#155724,color:#155724
    classDef control fill:#fff3cd,stroke:#856404,color:#856404
    classDef tools fill:#e2e3e5,stroke:#6c757d,color:#6c757d
    classDef external fill:#f8d7da,stroke:#721c24,color:#721c24
```
