How sastry-group/latte-mv works
Main Components and Modules
• Data Ingestion & Preprocessing:
– Identify the modules that handle raw video and sensor data. For example, BallTracker and PaddleTracker contain scripts (e.g., main.py, preprocess.py, clip.py) for handling input videos and extracting ball (and paddle) tracking data.
• 3D Reconstruction:
– HumanPoseTracker and RallyClipper are responsible for converting 2D detections (from ball, keypoints, table corners, etc.) into 3D reconstructions. They are configured with environments (via env.yml) and use external models (like PHALP for pose estimation).
• Anticipatory Model Training:
– The GenerativeModel module (with its own train.py, inference.py, and analysis scripts) is where the transformer is trained for anticipatory predictions. This model leverages data from the reconstructed outputs to predict ball trajectories.
• Simulation & Validation:
– MujocoRecon is used for validating the anticipatory predictions in a simulated environment. This phase uses MuJoCo for 3D physics simulation and validates the reconstructions using a robotic system.
– Represent this as a “Simulation & Evaluation” component that gets input from the anticipatory output.
• Visualization:
– Include a “Visualization” component linked to the output of both the anticipation and simulation modules.
• Supporting Utilities & Environment Setup:
– There are configuration files (env.yml, condaenv files) and build scripts that set up the experimental environments.
– Optionally show a “Configuration & Utilities” layer that supports all other modules.
Relationships and Data Flow
– From video input to preprocessing (BallTracker, PaddleTracker).
– From preprocessing to 3D reconstruction (HumanPoseTracker & RallyClipper).
– From reconstructed output to transformer training/inference (GenerativeModel).
– From predictions to simulation & evaluation (MujocoRecon).
– Back to visualization at multiple points in the pipeline.
– Pre-trained models and datasets (downloaded and later re-used) indicated alongside the corresponding module.
– Conda environments for each module showing the modularity/tracking of dependencies.
Architectural Patterns and Design Principles
• This is a staged pipeline, so illustrate the workflow as sequential stages rather than a monolithic architecture.
• Separation of concerns by isolating each phase into its submodule/directory.
– Data processing (e.g., ball and pose tracking modules)
– Model training and inference (the transformer module in GenerativeModel)
– Simulation (MuJoCo-based evaluation)
– Visualization and analysis.
Relevant Technologies, Frameworks, and Libraries
• Anaconda and Conda environments are central to managing dependencies. Note the use of environment files (env.yml, condaenv) across modules.
• MuJoCo is the key simulation library for Phase 3.
• Transformer-based models (for anticipatory prediction) are trained here.
• External libraries like PHALP for human pose estimation and perhaps TensorFlow/PyTorch (indirectly implied in training logs/events.out.tfevents files) should be mentioned.
• Pretrained models (downloaded via “gdown”) and their integration points.