How google-research/simclr works
Project type and purpose
• This is a machine-learning framework and CLI application for pretraining, fine-tuning, evaluating, and demoing SimCLR (contrastive learning) models in both TensorFlow 1.x and TensorFlow 2.x.
• It provides reusable modules (data pipelines, model definitions, loss functions, optimizers) and Colab notebooks to bootstrap research and demonstrations, as well as integration with Cloud TPUs and Google Cloud Storage for checkpointing.
Main components
a. Data Ingestion & Augmentation
• Modules: data.py, data_util.py
• Inputs: local datasets (CIFAR-10, TensorFlow Datasets ImageNet), imagenet_subsets/ text files
b. Model Construction
• Modules: resnet.py, model.py, model_util.py
• Components: ResNet base network (configurable depth/width/SK), projection head (configurable layers), supervised head for linear eval
c. Loss & Objective
• Module: objective.py
• Contrastive loss (NT-Xent), temperature hyperparameter
d. Optimizer
• Module: lars_optimizer.py
• Choices: Momentum or LARS, learning-rate schedules (linear or sqrt scaling), weight decay
e. Training & Evaluation Orchestration
• Entry point: run.py
• Modes: pretrain, finetune (linear head and full network), train_then_eval
• CLI flags → hyperparameter injection → initialize data, model, loss, optimizer → distribution strategy (TPU / single GPU) → training loop → evaluation loop
f. Logging & Checkpointing
• TensorBoard summaries
• Google Cloud Storage buckets for pretrained/fine-tuned/distilled models (gs://simclr-checkpoints/…)
g. Interactive Notebooks
• Folder: colabs/ (TF1.x) and tf2/colabs/ (TF2.x) – demonstration and tutorial notebooks for distillation, finetuning, inference, intriguing properties demos
Relationships & data flows
• The run.py orchestrator invokes the Data pipeline → feeds batched data into the Model → computes Loss → applies Optimizer → emits checkpoints and logs.
• Distribution strategy layer (tf.distribute.TPUStrategy or single-GPU) wraps the training loop.
• Checkpoint module reads/writes to GCS buckets; TensorBoard writes summaries to local or GCS paths.
• Colab notebooks import and call the same modules (data.py, run.py) for interactive demos.
Architectural patterns & design principles
• Modular layering: clear separation between data, model, objective, optimizer, and runner.
• CLI-driven configuration: all hyperparameters and modes controlled by flags to run.py.
• Dual support for TF1.x and TF2.x via parallel folder structures (root vs. tf2/).
• Use of tf.data for high-throughput data input and tf.distribute for distributed compute.
• Reusability and extensibility: easy to swap optimizers, change model depth, add new datasets or heads.
Technologies, frameworks & external services
• Core: Python, TensorFlow 1.x & 2.x, tf.data, tf.distribute, Keras (in TF2 path)
• Compute: Cloud TPU (via TPUName env + TPU strategy) or single GPU
• Storage: Google Cloud Storage (checkpoints), TensorFlow Datasets (ImageNet)
• Experiment tracking: TensorBoard
• Interactive demos: Jupyter/Colab notebooks
Essential elements
• Clear, consistent labels for every module and data store
• Internal modules vs. external services
• TF1.x vs. TF2.x implementation tracks