How maple-research-lab/caco works
Project Type and Purpose
• CaCo is a Python‑based self‑supervised contrastive‑learning framework built on PyTorch.
• Its goal is to train an encoder (e.g., ResNet‑50) by learning both positives and negatives cooperatively and adversarially via a memory module.
File‑Structure Analysis
• data_processing/: image loaders & augmentation (MultiCrop_Transform.py, imagenet loader)
• model/: network definitions (ResNet, CaCo memory bank) and optimizer wrappers
• ops/: environment config, argument parsing, LARS optimizer, k‑NN monitoring, OS utilities
• training/: distributed training orchestration (main_worker), memory initialization, training loops
• main.py / linear.py: high‐level entry points for pre‑training and linear evaluation
Key Insights from README
• Uses torch.distributed for multi‑GPU, multi‑node training
• Supports single‑crop and multi‑crop regimes
• Exposes knobs (learning rates, batch size, memory updates, cluster size) via CLI args
a. Data Storage
• ImageNet dataset on shared filesystem (NFS/HDFS/S3)
b. Data Processing Pipeline
• Loader → augmentation transforms (random crops, MultiCrop) → batch collator
c. Model & Memory Module
• Encoder backbone (ResNet)
• CaCo memory bank: learnable positive/negative samples
d. Loss & Optimizer
• Contrastive loss calculation (InfoNCE style)
• Optimizers: SGD + optional LARS (ops/LARS.py)
e. Training Orchestrator
• main.py invokes torch.distributed.launch → training/main_worker.py
• training/train_caco.py: forward, loss, backward, memory updates
f. Utilities & Monitoring
• knn_monitor for periodic evaluation
• tensorboardX logging, checkpointing
g. Compute Infrastructure
• GPU workers (8×V100 per node), multi‑node cluster
• torch.distributed backend (NCCL/TCP)
Relationships & Data Flow
• Data flows from Storage → data_processing.Loader → augment → Model
• Model outputs query/key embeddings → loss module consults Memory Bank → loss computed
• Gradient flow → Optimizer updates Model parameters and Memory Bank parameters
• Orchestrator coordinates DistributedDataParallel across GPUs/nodes
• knn_monitor reads current embeddings for online evaluation
• Logging & Checkpoint modules write to disk/remote storage
Architectural Patterns & Design Principles
• Modular separation: data, model, training, ops
• Pipeline pattern: sequential transforms → model → loss → optimizer → update
• Distributed Data Parallelism for scalability
• Config‐driven via ops/argparser.py and Config_Environment.py
Essential Elements Checklist
• Notes on parameters tuned via CLI (batch_size, lr, cluster_size, multi_crop flag)
• Where logs and checkpoints are written (disk path or remote storage)