# rafellerc/pytorch-siamfc architecture

> Architecture diagram of the GitHub repository rafellerc/pytorch-siamfc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rafellerc/pytorch-siamfc
- Repository: https://github.com/rafellerc/pytorch-siamfc

## Overview

1. **Repository purpose**
   - This is a PyTorch implementation of SiamFC-style object tracking, centered on learning a similarity function between a **reference patch** and a **search patch**.
   - The codebase is split into two practical products: **training/evaluation** and a **real-time visualization app**. Tracking itself is present as an application boundary, but the README says it is still under development.

2. **Top-level entrypoints**
   - `train.py` is the main orchestration script for **training, validation, restore-from-checkpoint, and evaluation-only mode**.
   - `vis_app.py` launches the visualization tool that runs the model over video sequences and overlays the score map in real time.
   - `setup.sh`, `profile_script.sh`, and `environment.yaml` support environment setup and profiling, not core model logic.

3. **Training pipeline boundary**
   - The `training/` package holds the core ML pipeline:
     - `datasets.py` and `crops_train.py` build training samples from video frames.
     - `labels.py` generates target correlation-map labels.
     - `models.py` defines the embedding networks used by the Siamese branch.
     - `losses.py`, `metrics.py`, and `optim.py` define objective, evaluation, and optimizer setup.
     - `train_utils.py` and `summary_utils.py` handle epoch loops, checkpointing, logging, and TensorBoard summaries.
   - The training flow is: **dataset frame pairs → crops/label map generation → embedding network → correlation output → loss/metrics → optimizer step**.

4. **Data source and sample construction**
   - Training uses the **ImageNet VID 2015** dataset, read from disk at runtime rather than preconverted.
   - The dataset logic appears designed to sample **reference/search pairs** from temporally separated frames, controlled by parameters such as `max_frame_sep`, `reference_sz`, `search_sz`, and `context_margin`.
   - The labels are spatial targets over the correlation map, using positive/negative thresholds (`pos_thr`, `neg_thr`) to create the classification map for each pair.

5. **Model architecture boundary**
   - `training/models.py` contains the embedding backbones used by SiamFC, including:
     - `BaselineEmbeddingNet`
     - `VGG11EmbeddingNet_5c`
     - `VGG16EmbeddingNet_8c`
   - The model is a **shared-weight Siamese embedding network**; the two branches process reference and search images separately, then a correlation operation produces the response map.
   - `BaselinePretrained.pth.tar` is the shipped pretrained checkpoint for the baseline model.

6. **Loss, metrics, and optimization**
   - `training/losses.py` defines the training objective, which the README describes as a binary cross-entropy-style loss over the correlation map.
   - `training/metrics.py` tracks training quality using:
     - **AUC**
     - **Center Error**
     - loss values
   - `training/optim.py` wires the optimizer, with support for at least **SGD** and **Adam** via parameters from the experiment config.

7. **Experiment/configuration model**
   - Training is organized around experiment folders under `training/experiments/`.
   - Each experiment stores a `parameters.json`, with `training/experiments/default/parameters.json` serving as the template.
   - This creates a clean boundary between **code** and **run configuration**, and each experiment folder becomes the unit of reproducibility and artifact storage.

8. **Runtime artifacts and logging**
   - `train.py` writes outputs into the experiment directory:
     - `train.log`
     - `metadata.train` / `metadata.val`
     - `metrics_val_last_weights.json` / `metrics_val_best_weights.json`
     - `best.pth.tar` / `last.pth.tar`
     - `tensorboard/` event data
   - That means the repo’s training architecture is not just compute; it also includes **persistent experiment state**, checkpointing, and summary generation.

9. **Inference/visualization application architecture**
   - `vis_app.py` uses a two-thread design implemented in `appSiamFC/`:
     - `producer.py` does the model inference and frame preparation.
     - `display.py` handles the GUI rendering.
     - `app_utils.py` contains shared application helpers.
   - The app computes a score map for each frame against a fixed reference embedding and overlays the result on the original video frame.

10. **Visualization data flow**
    - The visualization pipeline is:
      - load a sequence from the VID dataset
      - choose the first target frame as reference
      - resize so the reference region matches the expected network scale
      - compute the reference embedding once
      - for each frame, compute the full-frame score map
      - upscale and colorize the score map
      - display the overlay and center-error trace
    - The app is explicitly a **similarity-function explorer**, not a full tracker loop with target updates.

11. **Shared utility layer**
    - `utils/` is the cross-cutting support layer used by both training and visualization:
      - `bbox_transforms.py` and `crops.py` handle geometric conversions and crop extraction
      - `image_utils.py`, `visualization.py`, and `colormaps.py` support rendering
      - `load_baseline.py` loads the pretrained baseline weights
      - `tensor_conv.py` and `profiling.py` support core tensor ops and timing
      - `exceptions.py` centralizes repo-specific error types
    - The `utils/color_tables/` package provides colormap definitions used by the visualization app.

12. **External dependencies that affect the architecture**
    - The repo depends on **PyTorch** for model execution and training.
    - It uses **TensorBoard** for experiment summaries, **PyQt5/pyqtgraph** for the visualization GUI, and standard scientific Python packages for dataset and metric handling.
    - The architecture assumes access to a local copy of **ImageNet VID** on disk, so data ingestion is file-based rather than service-based.

13. **What to draw in an architecture diagram**
    - A useful diagram should show three main boundaries:
      - **Training pipeline**: dataset on disk → training package → checkpoint/metrics/tensorboard outputs
      - **Visualization app**: dataset + checkpoint → producer thread → display thread → GUI overlay
      - **Shared utilities**: geometry, image handling, plotting, checkpoint loading, and profiling
    - The model itself sits at the center as the shared Siamese embedding/correlation core, reused by both training and visualization.

Main entry points: [`appSiamFC/producer.py`](https://github.com/rafellerc/pytorch-siamfc/blob/master/appSiamFC/producer.py) (Producer), [`train.py`](https://github.com/rafellerc/pytorch-siamfc/blob/master/train.py) (train.py), [`training/train_utils.py`](https://github.com/rafellerc/pytorch-siamfc/blob/master/training/train_utils.py) (Train loop), [`training/models.py`](https://github.com/rafellerc/pytorch-siamfc/blob/master/training/models.py) (Siamese nets)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_training["Training pipeline"]
  node_train_py["train.py<br/>orchestrator<br/>[train.py]"]
  node_vid_data[("VID data<br/>dataset")]
  node_experiments["Experiments<br/>run config"]
  node_default_params["Default params<br/>json config<br/>[parameters.json]"]
  node_datasets_py["Dataset sampling<br/>data loader<br/>[datasets.py]"]
  node_crops_train_py["Train crops<br/>preprocess<br/>[crops_train.py]"]
  node_labels_py["Label maps<br/>target builder<br/>[labels.py]"]
  node_models_py["Siamese nets<br/>model<br/>[models.py]"]
  node_losses_py["Losses<br/>objective<br/>[losses.py]"]
  node_metrics_py["Metrics<br/>eval<br/>[metrics.py]"]
  node_optim_py["Optimizer<br/>[optim.py]"]
  node_train_utils_py["Train loop<br/>loop/checkpoint<br/>[train_utils.py]"]
  node_summary_utils_py["Summaries<br/>logging<br/>[summary_utils.py]"]
  node_best_ckpt[("Checkpoints<br/>artifacts")]
end

subgraph group_app["Visualization app"]
  node_vis_app_py["vis_app.py<br/>launcher<br/>[vis_app.py]"]
  node_producer_py["Producer<br/>inference thread<br/>[producer.py]"]
  node_display_py["Display<br/>gui thread<br/>[display.py]"]
  node_app_utils_py["App utils<br/>app helper<br/>[app_utils.py]"]
end

subgraph group_shared["Shared utilities"]
  node_baseline_ckpt[("Baseline weights<br/>checkpoint")]
  node_bbox_transforms_py["BBox transforms<br/>geometry<br/>[bbox_transforms.py]"]
  node_crops_py["Crops<br/>geometry<br/>[crops.py]"]
  node_image_utils_py["Image utils<br/>image ops<br/>[image_utils.py]"]
  node_visualization_py["Visualization<br/>rendering<br/>[visualization.py]"]
  node_colormaps_py["Colormaps<br/>palette<br/>[colormaps.py]"]
  node_load_baseline_py["Load baseline<br/>checkpoint loader<br/>[load_baseline.py]"]
  node_profiling_py["Profiling<br/>timing<br/>[profiling.py]"]
end

node_train_py -->|"reads/writes"| node_experiments
node_experiments -->|"template"| node_default_params
node_train_py -->|"loads"| node_vid_data
node_train_py -->|"builds batches"| node_datasets_py
node_datasets_py -->|"cropping"| node_crops_train_py
node_datasets_py -->|"targets"| node_labels_py
node_train_py -->|"forwards"| node_models_py
node_models_py -->|"response map"| node_losses_py
node_models_py -->|"evaluates"| node_metrics_py
node_train_py -->|"updates"| node_optim_py
node_train_py -->|"orchestrates"| node_train_utils_py
node_train_utils_py -->|"logs"| node_summary_utils_py
node_train_utils_py -->|"persists"| node_best_ckpt
node_train_utils_py -->|"restores"| node_baseline_ckpt
node_vis_app_py -->|"starts"| node_producer_py
node_vis_app_py -->|"starts"| node_display_py
node_producer_py -->|"loads model"| node_load_baseline_py
node_producer_py -->|"infers"| node_models_py
node_producer_py -->|"samples"| node_vid_data
node_producer_py -->|"maps boxes"| node_bbox_transforms_py
node_producer_py -->|"preprocesses"| node_crops_py
node_producer_py -->|"prepares frames"| node_image_utils_py
node_producer_py -->|"renders score"| node_visualization_py
node_display_py -->|"colors"| node_colormaps_py
node_display_py -->|"draws"| node_visualization_py
node_app_utils_py -->|"uses"| node_image_utils_py
node_train_py -->|"profiles"| node_profiling_py
node_vis_app_py -->|"profiles"| node_profiling_py

click node_train_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/train.py"
click node_vis_app_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/vis_app.py"
click node_experiments "https://github.com/rafellerc/pytorch-siamfc/tree/master/training/experiments"
click node_default_params "https://github.com/rafellerc/pytorch-siamfc/blob/master/training/experiments/default/parameters.json"
click node_datasets_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/training/datasets.py"
click node_crops_train_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/training/crops_train.py"
click node_labels_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/training/labels.py"
click node_models_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/training/models.py"
click node_losses_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/training/losses.py"
click node_metrics_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/training/metrics.py"
click node_optim_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/training/optim.py"
click node_train_utils_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/training/train_utils.py"
click node_summary_utils_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/training/summary_utils.py"
click node_baseline_ckpt "https://github.com/rafellerc/pytorch-siamfc/blob/master/BaselinePretrained.pth.tar"
click node_producer_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/appSiamFC/producer.py"
click node_display_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/appSiamFC/display.py"
click node_app_utils_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/appSiamFC/app_utils.py"
click node_bbox_transforms_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/utils/bbox_transforms.py"
click node_crops_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/utils/crops.py"
click node_image_utils_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/utils/image_utils.py"
click node_visualization_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/utils/visualization.py"
click node_colormaps_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/utils/colormaps.py"
click node_load_baseline_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/utils/load_baseline.py"
click node_profiling_py "https://github.com/rafellerc/pytorch-siamfc/blob/master/utils/profiling.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_train_py,node_vid_data,node_experiments,node_default_params,node_datasets_py,node_crops_train_py,node_labels_py,node_models_py,node_losses_py,node_metrics_py,node_optim_py,node_train_utils_py,node_summary_utils_py,node_best_ckpt toneBlue
class node_vis_app_py,node_producer_py,node_display_py,node_app_utils_py toneAmber
class node_baseline_ckpt,node_bbox_transforms_py,node_crops_py,node_image_utils_py,node_visualization_py,node_colormaps_py,node_load_baseline_py,node_profiling_py toneMint
```

## Components

### Training pipeline

Experiment-driven training, evaluation, and artifact generation.

- [train.py](https://github.com/rafellerc/pytorch-siamfc/blob/master/train.py): orchestrator · Main entrypoint for training, validation, restore, and eval-only runs. · `train.py`
- **VID data**: dataset · Local ImageNet VID 2015 frames on disk.
- [Experiments](https://github.com/rafellerc/pytorch-siamfc/tree/master/training/experiments): run config · Experiment folders with parameters and persisted artifacts. · `training/experiments`
- [Default params](https://github.com/rafellerc/pytorch-siamfc/blob/master/training/experiments/default/parameters.json): json config · Template hyperparameters for experiments. · `training/experiments/default/parameters.json`
- [Dataset sampling](https://github.com/rafellerc/pytorch-siamfc/blob/master/training/datasets.py): data loader · Builds reference/search pairs from video frames. · `training/datasets.py`
- [Train crops](https://github.com/rafellerc/pytorch-siamfc/blob/master/training/crops_train.py): preprocess · Extracts and normalizes training crops. · `training/crops_train.py`
- [Label maps](https://github.com/rafellerc/pytorch-siamfc/blob/master/training/labels.py): target builder · Creates correlation-map supervision with positive/negative thresholds. · `training/labels.py`
- [Siamese nets](https://github.com/rafellerc/pytorch-siamfc/blob/master/training/models.py): model · Shared-weight embedding backbones for reference and search branches. · `training/models.py`
- [Losses](https://github.com/rafellerc/pytorch-siamfc/blob/master/training/losses.py): objective · Binary cross-entropy-style objective over the response map. · `training/losses.py`
- [Metrics](https://github.com/rafellerc/pytorch-siamfc/blob/master/training/metrics.py): eval · Tracks AUC, center error, and loss. · `training/metrics.py`
- [Optimizer](https://github.com/rafellerc/pytorch-siamfc/blob/master/training/optim.py): optimizer setup · Configures SGD or Adam for training. · `training/optim.py`
- [Train loop](https://github.com/rafellerc/pytorch-siamfc/blob/master/training/train_utils.py): loop/checkpoint · Epoch orchestration, checkpointing, and run control. · `training/train_utils.py`
- [Summaries](https://github.com/rafellerc/pytorch-siamfc/blob/master/training/summary_utils.py): logging · TensorBoard and training summary writing. · `training/summary_utils.py`
- **Checkpoints**: artifacts · Best/last weights and validation metrics written per experiment.

### Visualization app

Real-time SiamFC similarity-map exploration over video frames.

- [vis_app.py](https://github.com/rafellerc/pytorch-siamfc/blob/master/vis_app.py): launcher · Starts the visualization application. · `vis_app.py`
- [Producer](https://github.com/rafellerc/pytorch-siamfc/blob/master/appSiamFC/producer.py): inference thread · Prepares frames and runs similarity-map inference. · `appSiamFC/producer.py`
- [Display](https://github.com/rafellerc/pytorch-siamfc/blob/master/appSiamFC/display.py): gui thread · Renders overlays, score maps, and traces. · `appSiamFC/display.py`
- [App utils](https://github.com/rafellerc/pytorch-siamfc/blob/master/appSiamFC/app_utils.py): app helper · Shared helper functions for the visualization app. · `appSiamFC/app_utils.py`

### Shared utilities

Common geometry, image, profiling, and checkpoint helpers used by both runtimes.

- [Baseline weights](https://github.com/rafellerc/pytorch-siamfc/blob/master/BaselinePretrained.pth.tar): checkpoint · Shipped pretrained baseline model weights. · `BaselinePretrained.pth.tar`
- [BBox transforms](https://github.com/rafellerc/pytorch-siamfc/blob/master/utils/bbox_transforms.py): geometry · Bounding-box coordinate conversions and resizing support. · `utils/bbox_transforms.py`
- [Crops](https://github.com/rafellerc/pytorch-siamfc/blob/master/utils/crops.py): geometry · Common crop extraction utilities. · `utils/crops.py`
- [Image utils](https://github.com/rafellerc/pytorch-siamfc/blob/master/utils/image_utils.py): image ops · Image loading, resizing, and rendering helpers. · `utils/image_utils.py`
- [Visualization](https://github.com/rafellerc/pytorch-siamfc/blob/master/utils/visualization.py): rendering · Plotting and overlay helpers for score maps. · `utils/visualization.py`
- [Colormaps](https://github.com/rafellerc/pytorch-siamfc/blob/master/utils/colormaps.py): palette · Color mapping utilities used by the GUI overlay. · `utils/colormaps.py`
- [Load baseline](https://github.com/rafellerc/pytorch-siamfc/blob/master/utils/load_baseline.py): checkpoint loader · Loads pretrained baseline weights for inference or initialization. · `utils/load_baseline.py`
- [Profiling](https://github.com/rafellerc/pytorch-siamfc/blob/master/utils/profiling.py): timing · Runtime measurement helpers. · `utils/profiling.py`

## Connections

- train.py → Experiments: reads/writes · Uses experiment folders as the unit of configuration and artifact storage.
- Experiments → Default params: template · Default parameters seed new experiments.
- train.py → VID data: loads · Consumes ImageNet VID frames from disk.
- train.py → Dataset sampling: builds batches · Creates temporal reference/search pairs.
- Dataset sampling → Train crops: cropping · Generates training crops around sampled frames.
- Dataset sampling → Label maps: targets · Builds response-map supervision.
- train.py → Siamese nets: forwards · Runs the Siamese embedding network.
- Siamese nets → Losses: response map · Correlation output is scored by the training loss.
- Siamese nets → Metrics: evaluates · Prediction quality is measured with AUC and center error.
- train.py → Optimizer: updates · Configures optimizer steps.
- train.py → Train loop: orchestrates · Main training loop and checkpoint control.
- Train loop → Summaries: logs · Writes TensorBoard summaries and run logs.
- Train loop → Checkpoints: persists · Stores best and last weights plus validation metrics.
- Train loop → Baseline weights: restores · Can initialize or resume from pretrained weights.
- vis_app.py → Producer: starts · Launches the inference side of the app.
- vis_app.py → Display: starts · Launches the rendering side of the app.
- Producer → Load baseline: loads model · Initializes the baseline checkpoint for inference.
- Producer → Siamese nets: infers · Computes the similarity response map.
- Producer → VID data: samples · Reads video frames for visualization.
- Producer → BBox transforms: maps boxes · Transforms regions between frame and network space.
- Producer → Crops: preprocesses · Extracts reference and search crops.
- Producer → Image utils: prepares frames · Resizes and formats frames for display and inference.
- Producer → Visualization: renders score · Produces overlays and heatmaps.
- Display → Colormaps: colors · Applies palettes to the response map overlay.
- Display → Visualization: draws · Uses shared plotting and overlay helpers.
- App utils → Image utils: uses · Shares image-handling helpers with the display pipeline.
- train.py → Profiling: profiles · Supports timing and profiling during runs.
- vis_app.py → Profiling: profiles · Supports timing during visualization runs.