How rafellerc/pytorch-siamfc works
1. Repository purpose
- This is a PyTorch implementation of SiamFC-style object tracking, centered on learning a similarity function between a reference patch and a search patch.
- The codebase is split into two practical products: training/evaluation and a real-time visualization app. Tracking itself is present as an application boundary, but the README says it is still under development.
2. Top-level entrypoints
- train.py is the main orchestration script for training, validation, restore-from-checkpoint, and evaluation-only mode.
- vis_app.py launches the visualization tool that runs the model over video sequences and overlays the score map in real time.
- setup.sh, profile_script.sh, and environment.yaml support environment setup and profiling, not core model logic.
3. Training pipeline boundary
- The training/ package holds the core ML pipeline:
- datasets.py and crops_train.py build training samples from video frames.
- labels.py generates target correlation-map labels.
- models.py defines the embedding networks used by the Siamese branch.
- losses.py, metrics.py, and optim.py define objective, evaluation, and optimizer setup.
- train_utils.py and summary_utils.py handle epoch loops, checkpointing, logging, and TensorBoard summaries.
- The training flow is: dataset frame pairs → crops/label map generation → embedding network → correlation output → loss/metrics → optimizer step.
4. Data source and sample construction
- Training uses the ImageNet VID 2015 dataset, read from disk at runtime rather than preconverted.
- The dataset logic appears designed to sample reference/search pairs from temporally separated frames, controlled by parameters such as max_frame_sep, reference_sz, search_sz, and context_margin.
- The labels are spatial targets over the correlation map, using positive/negative thresholds (pos_thr, neg_thr) to create the classification map for each pair.
5. Model architecture boundary
- training/models.py contains the embedding backbones used by SiamFC, including:
- BaselineEmbeddingNet
- VGG11EmbeddingNet_5c
- VGG16EmbeddingNet_8c
- The model is a shared-weight Siamese embedding network; the two branches process reference and search images separately, then a correlation operation produces the response map.
- BaselinePretrained.pth.tar is the shipped pretrained checkpoint for the baseline model.
6. Loss, metrics, and optimization
- training/losses.py defines the training objective, which the README describes as a binary cross-entropy-style loss over the correlation map.
- training/metrics.py tracks training quality using:
- AUC
- Center Error
- loss values
- training/optim.py wires the optimizer, with support for at least SGD and Adam via parameters from the experiment config.
7. Experiment/configuration model
- Training is organized around experiment folders under training/experiments/.
- Each experiment stores a parameters.json, with training/experiments/default/parameters.json serving as the template.
- This creates a clean boundary between code and run configuration, and each experiment folder becomes the unit of reproducibility and artifact storage.
8. Runtime artifacts and logging
- train.py writes outputs into the experiment directory:
- train.log
- metadata.train / metadata.val
- metrics_val_last_weights.json / metrics_val_best_weights.json
- best.pth.tar / last.pth.tar
- tensorboard/ event data
- That means the repo’s training architecture is not just compute; it also includes persistent experiment state, checkpointing, and summary generation.
9. Inference/visualization application architecture
- vis_app.py uses a two-thread design implemented in appSiamFC/:
- producer.py does the model inference and frame preparation.
- display.py handles the GUI rendering.
- app_utils.py contains shared application helpers.
- The app computes a score map for each frame against a fixed reference embedding and overlays the result on the original video frame.
10. Visualization data flow
- The visualization pipeline is:
- load a sequence from the VID dataset
- choose the first target frame as reference
- resize so the reference region matches the expected network scale
- compute the reference embedding once
- for each frame, compute the full-frame score map
- upscale and colorize the score map
- display the overlay and center-error trace
- The app is explicitly a similarity-function explorer, not a full tracker loop with target updates.
11. Shared utility layer
- utils/ is the cross-cutting support layer used by both training and visualization:
- bbox_transforms.py and crops.py handle geometric conversions and crop extraction
- image_utils.py, visualization.py, and colormaps.py support rendering
- load_baseline.py loads the pretrained baseline weights
- tensor_conv.py and profiling.py support core tensor ops and timing
- exceptions.py centralizes repo-specific error types
- The utils/color_tables/ package provides colormap definitions used by the visualization app.
12. External dependencies that affect the architecture
- The repo depends on PyTorch for model execution and training.
- It uses TensorBoard for experiment summaries, PyQt5/pyqtgraph for the visualization GUI, and standard scientific Python packages for dataset and metric handling.
- The architecture assumes access to a local copy of ImageNet VID on disk, so data ingestion is file-based rather than service-based.
13. What to draw in an architecture diagram
- A useful diagram should show three main boundaries:
- Training pipeline: dataset on disk → training package → checkpoint/metrics/tensorboard outputs
- Visualization app: dataset + checkpoint → producer thread → display thread → GUI overlay
- Shared utilities: geometry, image handling, plotting, checkpoint loading, and profiling
- The model itself sits at the center as the shared Siamese embedding/correlation core, reused by both training and visualization.
Main entry points: producer.py Producer · train.py train.py · train_utils.py Train loop · models.py Siamese nets
Components 26
Training pipeline
Experiment-driven training, evaluation, and artifact generation.
- train.py · Main entrypoint for training, validation, restore, and eval-only runs.
train.py - VID data · Local ImageNet VID 2015 frames on disk.
- Experiments · Experiment folders with parameters and persisted artifacts.
training/experiments - Default params · Template hyperparameters for experiments.
training/experiments/default/parameters.json - Dataset sampling · Builds reference/search pairs from video frames.
training/datasets.py - Train crops · Extracts and normalizes training crops.
training/crops_train.py - Label maps · Creates correlation-map supervision with positive/negative thresholds.
training/labels.py - Siamese nets · Shared-weight embedding backbones for reference and search branches.
training/models.py - Losses · Binary cross-entropy-style objective over the response map.
training/losses.py - Metrics · Tracks AUC, center error, and loss.
training/metrics.py - Optimizer · Configures SGD or Adam for training.
training/optim.py - Train loop · Epoch orchestration, checkpointing, and run control.
training/train_utils.py - Summaries · TensorBoard and training summary writing.
training/summary_utils.py - Checkpoints · Best/last weights and validation metrics written per experiment.
Visualization app
Real-time SiamFC similarity-map exploration over video frames.
- vis_app.py · Starts the visualization application.
vis_app.py - Producer · Prepares frames and runs similarity-map inference.
appSiamFC/producer.py - Display · Renders overlays, score maps, and traces.
appSiamFC/display.py - App utils · Shared helper functions for the visualization app.
appSiamFC/app_utils.py
Shared utilities
Common geometry, image, profiling, and checkpoint helpers used by both runtimes.
- Baseline weights · Shipped pretrained baseline model weights.
BaselinePretrained.pth.tar - BBox transforms · Bounding-box coordinate conversions and resizing support.
utils/bbox_transforms.py - Crops · Common crop extraction utilities.
utils/crops.py - Image utils · Image loading, resizing, and rendering helpers.
utils/image_utils.py - Visualization · Plotting and overlay helpers for score maps.
utils/visualization.py - Colormaps · Color mapping utilities used by the GUI overlay.
utils/colormaps.py - Load baseline · Loads pretrained baseline weights for inference or initialization.
utils/load_baseline.py - Profiling · Runtime measurement helpers.
utils/profiling.py
Connections 28
- train.py → Experiments: reads/writes · Uses experiment folders as the unit of configuration and artifact storage.
- Experiments → Default params: template · Default parameters seed new experiments.
- train.py → VID data: loads · Consumes ImageNet VID frames from disk.
- train.py → Dataset sampling: builds batches · Creates temporal reference/search pairs.
- Dataset sampling → Train crops: cropping · Generates training crops around sampled frames.
- Dataset sampling → Label maps: targets · Builds response-map supervision.
- train.py → Siamese nets: forwards · Runs the Siamese embedding network.
- Siamese nets → Losses: response map · Correlation output is scored by the training loss.
- Siamese nets → Metrics: evaluates · Prediction quality is measured with AUC and center error.
- train.py → Optimizer: updates · Configures optimizer steps.
- train.py → Train loop: orchestrates · Main training loop and checkpoint control.
- Train loop → Summaries: logs · Writes TensorBoard summaries and run logs.
- Train loop → Checkpoints: persists · Stores best and last weights plus validation metrics.
- Train loop → Baseline weights: restores · Can initialize or resume from pretrained weights.
- vis_app.py → Producer: starts · Launches the inference side of the app.
- vis_app.py → Display: starts · Launches the rendering side of the app.
- Producer → Load baseline: loads model · Initializes the baseline checkpoint for inference.
- Producer → Siamese nets: infers · Computes the similarity response map.
- Producer → VID data: samples · Reads video frames for visualization.
- Producer → BBox transforms: maps boxes · Transforms regions between frame and network space.
- Producer → Crops: preprocesses · Extracts reference and search crops.
- Producer → Image utils: prepares frames · Resizes and formats frames for display and inference.
- Producer → Visualization: renders score · Produces overlays and heatmaps.
- Display → Colormaps: colors · Applies palettes to the response map overlay.
- Display → Visualization: draws · Uses shared plotting and overlay helpers.
- App utils → Image utils: uses · Shares image-handling helpers with the display pipeline.
- train.py → Profiling: profiles · Supports timing and profiling during runs.
- vis_app.py → Profiling: profiles · Supports timing during visualization runs.