How sfig611/tabbie works
Project Type & Purpose
• Tabbie is a machine‐learning framework for tabular information embedding.
• It supports a pre‐training stage (corrupt‐cell detection) and several fine‐tuning tasks (cell, column, row, table classification).
Top‐Level Components
A. Data Sources (blue)
• Pretraining JSONL (“data/pretrain/sample.jsonl”)
• CSV directories & labels for fine‐tuning:
– data/ft_cell/{train,test,valid}_csv + train_label.csv, valid_label.csv, test_csv + test_label.csv
– data/ft_col/...
– data/ft_table/...
B. Environment & Dependencies (light gray)
• Conda environment (env/env.yml) with PyTorch + CUDA + apex
• Pretrained model archives (model/freq.tar.gz, mix.tar.gz)
• BERT vocab (data/bert‐base‐uncased‐vocab.txt)
C. Preprocessing & Feature Extraction (green)
• scripts/to_npy.py, ext_feat.py – convert CSV to npy arrays, external features
• scripts/bert.py – tokenization, vocabulary lookup
• scripts/col_pop.py, row_pop.py, sato.py – specialized feature pipelines
D. Data Readers (yellow)
• table_embedder/readers – task‐specific readers (cell_reader, cp_reader, rp_reader, sato_reader, pretrain_reader)
• epoch callbacks (write_callback)
E. Model Library (orange)
• table_embedder/models/lib – core building blocks (BERT token embedder, self‐attention layers, masked FF)
• table_embedder/models/pretrain.py – pretraining architecture
• table_embedder/models/finetune_{cell,col,cp,rp,sato,table}.py – fine‐tuning heads
F. Training & Experiment Management (purple)
• CLI entrypoints:
– pretrain: scripts/pretrain.py
– fine‐tuning: train.py
• exp/…/*.jsonnet + *.yml – experiment configurations (hyperparameters, schedules)
• Uses Apex for mixed‐precision
G. Model Artifacts & Output Storage (brown)
• out/ – output model directory (out_model/)
• model/ – pretrained archives
• logs, predictions, saved weights
H. Prediction Pipeline (teal)
• pretrain_pred.py – corrupt‐cell detection inference
• pred.py – fine‐tuned task inference
• table_embedder/predictors – predictor classes (cell_predictor, generic predictor)
Relationships & Data Flow
1. Raw Data → Preprocessing Scripts → Preprocessed Files (npy, token IDs)
2. Preprocessed Files → Data Readers → Torch Datasets & DataLoaders
3. Datasets → Model Library → Training Loop (train.py / scripts/pretrain.py)
4. Training Loop ↔ Experiment Config (exp/*.yml)
5. Training Loop → Model Artifacts (out_model/)
6. Model Artifacts + Test Data → Prediction Scripts → Final Predictions
Architectural Patterns & Design Principles
• Modular “reader → model → predictor” pipeline for each task
• Separation of concerns:
• scripts/ → orchestration & utilities
• table_embedder/ → core library (IO, model, prediction)
• exp/ → configuration management
• Use of config‐as‐code (jsonnet + YAML) for reproducibility
• Plugin‐style readers & predictors to handle different tasks
Elements
• Data Sources
• Preprocessing & Feature Extraction
• Data Readers
• Model Library
• Training & Experiment Manager
• Model Artifacts
• Prediction Pipeline
• Call out config files & environment dependencies on the side