How soda-inria/tabicl works
1. Package shape and public entrypoints
- This is a Python library, not a service. The public API is exposed from src/tabicl/__init__.py and is centered on a few user-facing classes: TabICLClassifier, TabICLRegressor, FinetunedTabICLClassifier, FinetunedTabICLRegressor, TabICLForecaster, plus SHAP helpers.
- The repository is organized into internal subpackages for inference/modeling, sklearn integration, finetuning, forecasting, interpretability, and pretraining.
2. Core inference path: tabular in-context learning
- The main runtime flow is fit(X_train, y_train) -> predict(X_test) on a pretrained transformer.
- fit() mainly stores and preprocesses the training context; predict() performs the actual in-context forward pass through the model using (X_train, y_train, X_test).
- This is the central architectural boundary: user tabular data enters, becomes model-ready tensors, then a pretrained transformer emits class probabilities or regression outputs.
3. Model subsystem
- src/tabicl/_model contains the Transformer-style architecture and supporting layers: embeddings, attention, encoders, interaction blocks, quantile/distribution heads, rotary embeddings, and learning/inference utilities.
- tabicl.py likely assembles the end-to-end model used by the high-level estimators.
- inference.py, inference_config.py, and kv_cache.py capture inference-time controls such as checkpoint loading, caching, and memory/offload behavior.
- This subsystem is where the pretrained checkpoints are consumed.
4. Preprocessing and tabular normalization
- src/tabicl/_sklearn/preprocessing.py and related utilities handle the tabular input boundary for both pandas DataFrames and NumPy arrays.
- The library performs categorical detection/ordinal encoding, missing-value handling, numerical imputation, clipping/outlier handling, scaling/normalization, and feature shuffling for ensemble diversity.
- This preprocessing layer is important because it bridges heterogeneous user data formats to the model’s fixed tensor expectations.
5. Scikit-learn compatibility layer
- src/tabicl/_sklearn wraps the model in sklearn-style estimators and ensures it can be used in pipelines.
- base.py, classifier.py, regressor.py, and sklearn_utils.py provide the fit/predict API, parameter handling, and serialization behavior expected from sklearn estimators.
- This is the main integration boundary for external ML workflows; the repo is designed to plug into sklearn pipelines rather than expose a custom training framework only.
6. Checkpointing, persistence, and runtime configuration
- The README’s save() / load() behavior maps to estimator-level persistence, with options to save model weights, training data, and KV cache.
- Model weights can be downloaded from Hugging Face or loaded from a local model_path; this makes pretrained checkpoints an external dependency of the inference path.
- Runtime knobs such as device selection, mixed precision, Flash Attention 3, offloading, and batch sizing all affect inference execution but do not change the logical model flow.
7. KV cache and scalability
- kv_cache.py and the README’s cache option show a distinct performance subsystem: training-context projections can be cached during fit() and reused across repeated predict() calls.
- The architecture therefore has two inference modes: fresh full-context inference, and cached-context inference.
- Offloading support (CPU/disk) is part of the same scalability boundary and matters for large datasets or constrained memory environments.
8. Fine-tuning subsystem
- src/tabicl/_finetune provides a separate training loop for adapting the pretrained model to a single downstream dataset.
- It includes dataset preparation, logging, classifier/regressor wrappers, and a base training workflow.
- The README indicates this path uses PyTorch training utilities such as AdamW, cosine warmup, gradient clipping, early stopping, and multi-GPU via torchrun.
- Architecturally, this is distinct from zero-shot inference: instead of pure forward-pass prediction, it updates model weights and writes a new checkpoint.
9. Forecasting subsystem
- src/tabicl/forecast reuses TabICL as the predictive backend for time series forecasting.
- It includes a forecasting engine, dispatch logic, a forecaster wrapper, time-series dataframe abstraction, preprocessing, plotting, and transform pipelines for calendar/seasonality features.
- The flow is: time series data is normalized into a TimeSeriesDataFrame, transformed into tabular context, passed through a TabICL regressor-backed forecaster, then optionally visualized.
- This subsystem is a domain-specific adapter around the core tabular model.
10. Interpretability subsystem
- src/tabicl/shap integrates SHAP with TabICL.
- The key architectural choice is using an all-NaN background row so that feature masking aligns with TabICL’s native missing-value handling instead of substituting arbitrary reference values.
- This sits outside the core prediction path but consumes a fitted estimator and input samples to produce attribution outputs.
11. Pretraining / synthetic data generation
- src/tabicl/prior contains the pretraining-side machinery: dataset generation/loading, prior configuration, hyperparameter sampling, synthetic SCMs (MLP/tree), activation utilities, and regression-to-classification helpers.
- The scripts in scripts/train_stage1.sh, train_stage2.sh, and train_stage3.sh indicate a staged curriculum-learning pipeline for v1 pretraining.
- This subsystem is upstream of the shipped checkpoints and is separate from end-user inference.
12. Training orchestration for pretraining
- src/tabicl/train provides training configuration, optimization, and run orchestration for the pretraining workflow.
- It is distinct from _finetune: pretraining operates on generated/synthetic data and curriculum stages, while finetuning adapts to a specific real dataset.
- Together, these two training subsystems form the model-development side of the repository.
13. Tests, docs, and tutorials as support layers
- tests/ focuses on API compatibility and input handling, especially NumPy inputs and string/categorical data.
- tutorials/ demonstrates the main architectural use cases: classification, regression, fine-tuning, interpretability, forecasting, string handling, and unsupervised learning.
- docs/ and CI workflows are support infrastructure; they document and validate the library but are not part of runtime execution.
14. Overall architecture in one line
- The repo is a pretrained-transformer tabular learning platform with a shared core model, thin sklearn-compatible estimators, optional fine-tuning, domain adapters for forecasting and SHAP, and a separate synthetic-data pretraining pipeline that produced the distributed checkpoints.
Main entry points: base.py Base estimator · tabicl.py TabICL model · inference.py Inference · __init__.py tabicl
Components 20
Public API
User-facing imports and estimator entrypoints.
- tabicl · Top-level module exporting the main estimators and helpers.
src/tabicl/__init__.py - Classifier API · High-level tabular classification wrapper.
src/tabicl/_sklearn/classifier.py - Regressor API · High-level tabular regression wrapper.
src/tabicl/_sklearn/regressor.py
Core model
Pretrained transformer, inference, and cache behavior.
- TabICL model · End-to-end pretrained tabular in-context learning model assembly.
src/tabicl/_model/tabicl.py - Model blocks · Embeddings, attention, encoders, interaction blocks, and heads.
src/tabicl/_model/layers.py - Inference · Checkpoint loading, forward-pass execution, and inference-time control flow.
src/tabicl/_model/inference.py - Inference cfg · Device, precision, offload, and batch-size settings for inference.
src/tabicl/_model/inference_config.py - KV cache · Cached training-context projections reused across repeated predictions.
src/tabicl/_model/kv_cache.py
Sklearn layer
Tabular preprocessing and sklearn-compatible estimators.
- Base estimator · Shared fit/predict lifecycle, persistence, and model wiring.
src/tabicl/_sklearn/base.py - Preprocessing · Input normalization for pandas/NumPy, categorical handling, imputation, scaling, and shuffling.
src/tabicl/_sklearn/preprocessing.py - Sklearn utils · Shared estimator utilities and parameter/serialization helpers.
src/tabicl/_sklearn/sklearn_utils.py
Fine-tuning
Dataset adaptation and weight-updating training loop.
- Finetune base · Shared downstream fine-tuning workflow and checkpoint writing.
src/tabicl/_finetune/base.py - Finetune data · Dataset preparation and batching for downstream adaptation.
src/tabicl/_finetune/data.py
Forecasting
Time-series adapter built on the core regressor.
- Forecaster · Time-series wrapper that routes forecasting through TabICL.
src/tabicl/forecast/_forecaster.py - TS frame · Time-series dataframe abstraction used by the forecasting pipeline.
src/tabicl/forecast/_ts_dataframe.py
Interpretability
SHAP-based attribution utilities.
- SHAP · Feature attribution integration aligned with TabICL missing-value behavior.
src/tabicl/shap/_shap.py
Model development
Synthetic pretraining and training orchestration for released checkpoints.
- Prior pipeline · Synthetic dataset generation, priors, and curriculum data sources.
src/tabicl/prior/_dataset.py - Train run · Pretraining orchestration, optimization, and staged run control.
src/tabicl/train/_run.py
Support
Unsupervised helpers and tests that exercise the public API.
- Unsupervised · Unsupervised learning entrypoint built on the shared library.
src/tabicl/_unsupervised/unsupervised.py - Tests · Repository tests covering NumPy, strings, and sklearn behavior.
tests/test_sklearn.py
Connections 22
- tabicl → Classifier API: exports · Top-level module exposes classifier entrypoints.
- tabicl → Regressor API: exports · Top-level module exposes regressor entrypoints.
- Classifier API → Base estimator: uses · Classifier wraps the shared sklearn base lifecycle.
- Regressor API → Base estimator: uses · Regressor wraps the shared sklearn base lifecycle.
- Base estimator → Preprocessing: prepares data · Fit/predict boundaries normalize heterogeneous tabular inputs.
- Base estimator → Sklearn utils: depends on · Shared estimator utilities support parameters and persistence.
- Base estimator → TabICL model: loads model · Estimator binds to the pretrained TabICL backbone.
- TabICL model → Model blocks: assembled from · Model composition is built from embeddings, attention, encoders, and heads.
- TabICL model → Inference: served by · Inference module executes the model with loaded checkpoints.
- Inference → Inference cfg: configured by · Runtime controls select precision, device, batching, and offload.
- Inference → KV cache: reuses · Cached context is reused for repeated predictions.
- Base estimator → KV cache: optionally persists · Estimator-level save/load can include cached context.
- Finetune base → Finetune data: trains on · Fine-tuning uses adapted downstream datasets and loaders.
- Finetune base → TabICL model: updates · Fine-tuning adjusts pretrained weights to a single dataset.
- Forecaster → TS frame: normalizes · Forecasting begins by converting series data into a time-series frame.
- Forecaster → Regressor API: delegates to · Forecasting reuses the regression pathway as its predictive backend.
- SHAP → Base estimator: explains · Attribution utilities consume fitted estimators and samples.
- Prior pipeline → Train run: feeds · Synthetic data and priors drive the staged pretraining workflow.
- Train run → TabICL model: produces · Pretraining writes checkpoints consumed by inference.
- Unsupervised → Base estimator: shares core · Unsupervised mode reuses the common library stack.
- Tests → tabicl: validates · Tests exercise the public API and input handling.
- Tests → Preprocessing: covers · Tests check NumPy and string/categorical preprocessing paths.