How jeffacce/dynamo_ssl works
This project is a research–oriented visual pretraining system for visuo-motor control. It integrates dataset management, simulation environments, deep learning model training, and evaluation components into one coherent suite. The system’s aim is to pretrain visual encoders (using methods such as dynamics pretraining and variants of self‐supervised learning) and then test them in several simulated robotic environments (e.g. sim kitchen, block pushing, LIBERO Goal, Push-T).
Main Components
• Environment and Simulation Modules:
– The “envs” directory has multiple simulation environments (e.g. sim_kitchen, block pushing, libero, pusht). Also include the assets, physics simulation wrappers, robots, and associated BDDL files.
• Dataset Management:
– “datasets” contains modules for various dataset loaders (including a template “your_dataset.py”). This is connected to configuration files that define dataset paths via environment variables.
• Configuration System:
– The “configs” and “eval_configs” directories hold YAML configuration files for defining environment settings, training parameters, SSL options, inverse dynamics/projector settings, and encoder architectures. They drive the behavior of both training and evaluation pipelines.
• Models and Architectures:
– The “models” module encapsulates the visual encoder (with variants like ResNet and multiview stacking), projector models, self-supervised learning modules (“ssl”), and also the transformer components (e.g. the vq_behavior_transformer). These represent the “brains” of the system.
• Training and Evaluation Pipelines:
– The “train.py” script is the entry point for training the encoder models.
– The “online_eval.py” script is used for evaluating the trained model snapshots in simulation.
• Workspaces:
– The “workspaces” folder contains experiment-specific code that handles both training diagnostics (linear probes, nearest neighbor mappings) and other per-environment pipelines.
• Utilities and Supporting Scripts:
– The “utils” package and additional scripts (e.g., “online_eval.py”) provide key functionalities, such as inference, logging (with wandb), video creation, and sometimes dataset preparation.
Relationships and Data Flow
• Datasets feeding data into training pipelines, which in turn supply batches to the models.
• Models with the “train.py” training loop, clearly indicating that the deep learning encoder along with auxiliary (projector and SSL) components are being optimized using the data.
• Additional utilities like logging with wandb and video generation that ties back to both training and evaluation modules.
Architectural Patterns and Technologies
• Modular Design – The system is decomposed into well‐separated modules by function (dataset handling, simulation, configuration, training, evaluation, and models), which is typical for full research pipelines.
• Config-Driven Execution – The heavy use of YAML configuration files means the system design should emphasize a central “configuration hub” that governs both training and evaluation.
• Technologies/Libraries – Emphasize the use of Conda environments (conda_env.yml), wandb for logging, and potentially PyTorch (implied in model modules) and simulation libraries (MuJoCo, PyBullet or similar for sim environments) even though not explicitly mentioned.
Additional Detailed Elements
• Separation between training and evaluation workflows.
• Incorporate the concept of “workspaces” as experimental setups that combine data passes, diagnostics, and model evaluations.
• Optionally, indicate common model pipeline components: an encoder branch, a projection head (inverse dynamics, identity, etc.), and an optional EMA or SimSiam branch (as mentioned in training configs for your dataset).