How jose-melo/t-jepa works
Project Type and Purpose
• This is an experimental deep learning framework focused on self-supervised learning for tabular data (an augmentation‐free Joint Embedding Predictive Architecture). Its main goal is to predict subsets of latent representations without data augmentations, thereby yielding competitive performance compared to standard gradient‐boosted machine learning techniques.
• The README and file structure show that the project supports both training (via run.py and launch_tjepa.sh) and benchmarking (via benchmark.py and numerous configuration files) among several models (e.g., AutoInt, DCNv2, FT-Transformer, MLP, ResNet) across different datasets.
Analyzing the File Structure
• Top-level directories:
• “src”: Contains all core source code with key components (datasets, models, training scripts, utility functions, and the T-JEPA-specific transformer).
• “datasets”: Holds data loading and preprocessing code.
• “benchmark” (both at the top level and under src/benchmark): Contains configuration files, model configurations, and scripts for running experiments.
• “scripts”: Provides shell scripts to launch training, benchmark runs, tuning, and experiments with various random seeds.
• “results”: Organized per dataset and per model, storing JSON outputs and summary files.
• Multiple configuration subdirectories (e.g., tuned_config, tuning_config) reside under src/benchmark and ensure decoupled configuration management for each experiment.
• Other key files include run.py (main entry for training/evaluation), benchmark.py (to run benchmarking experiments), and launch_tjepa.sh (to start pretraining).
Key Architectural Components
• Data Ingestion & Preprocessing:
• External datasets (downloaded via download_data.sh) and the “datasets” module that handles loading and embedded representations (e.g., adult_income.py, aloi.py).
• Core Training Pipeline:
• T-JEPA Learning Flow:
• Context Encoder
• Target Encoder
• Prediction Module (integrated in src/tjepa_transformer.py)
• Training and evaluation are orchestrated by src/train.py and integrated into the main run.py.
• Model Components:
• Benchmarking & Evaluation:
• Benchmarking scripts and configurations (benchmark.py, run_benchmark.py, various JSON config files under src/benchmark/tuned_config) that test and compare models across datasets.
• Configuration and Utility Modules:
• Config files (configs.py) and utilities (inside src/utils) like logging, checkpointing, and optimization functions.
• Experiment Orchestration:
• Shell scripts in the “scripts” folder (launch_tjepa.sh, bench_random_seed, tjepa_tuning) that allow parameter tuning and experiment runs.
• Results Storage:
• The “results” directory tree where outputs (JSON files and summaries) for different models and datasets are stored.
How They Interact
• The training process (triggered by run.py or launch_tjepa.sh) acquires data from the datasets folder, processes it via the data ingestion modules, and feeds it into the T-JEPA pipeline composed of the context encoder, target encoder, and prediction module.
• Model definitions (in src/models) are called by the training routines; these return latent representations used in predictions.
• Utility modules support training (e.g., logging, model saving, learning rate scheduling) and are linked with train_utils.
• Benchmarking scripts (benchmark.py, run_benchmark.py) leverage tuned configuration files to run experiments across multiple models and datasets, then save results into the “results” directory.
• The configuration (from both JSON and YAML files) decouples model hyperparameters and training regimes, ensuring flexibility in experiments.
Architectural Patterns and Design Principles
• Modularity: Each functionality (data loading, model definition, training loop, utility, benchmarking) is isolated in its own subdirectory.
• Configurability: Files under tuned_config and tuning_config separate experiment parameters from code.
• Extensibility: The clear separation between model implementations (each in its own file under src/models) and the overall pipeline (e.g., in src/train.py).
• Reproducibility: Scripts in “bench_random_seed” indicate careful logging of experiments and random seed management.