# sfig611/tabbie architecture

> Architecture diagram of the GitHub repository sfig611/tabbie, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sfig611/tabbie
- Repository: https://github.com/sfig611/tabbie

## Overview

##### Project Type & Purpose
- Tabbie is a machine‐learning framework for tabular information embedding.
- It supports a pre‐training stage (corrupt‐cell detection) and several fine‐tuning tasks (cell, column, row, table classification).

##### Top‐Level Components
A. Data Sources (blue)
- Pretraining JSONL (“data/pretrain/sample.jsonl”)
- CSV directories & labels for fine‐tuning:
– data/ft_cell/{train,test,valid}_csv + train_label.csv, valid_label.csv, test_csv + test_label.csv
– data/ft_col/...
– data/ft_table/...

B. Environment & Dependencies (light gray)
- Conda environment (env/env.yml) with PyTorch + CUDA + apex
- Pretrained model archives (model/freq.tar.gz, mix.tar.gz)
- BERT vocab (data/bert‐base‐uncased‐vocab.txt)

C. Preprocessing & Feature Extraction (green)
- scripts/to_npy.py, ext_feat.py – convert CSV to npy arrays, external features
- scripts/bert.py – tokenization, vocabulary lookup
- scripts/col_pop.py, row_pop.py, sato.py – specialized feature pipelines

D. Data Readers (yellow)
- table_embedder/readers – task‐specific readers (cell_reader, cp_reader, rp_reader, sato_reader, pretrain_reader)
- epoch callbacks (write_callback)

E. Model Library (orange)
- table_embedder/models/lib – core building blocks (BERT token embedder, self‐attention layers, masked FF)
- table_embedder/models/pretrain.py – pretraining architecture
- table_embedder/models/finetune_{cell,col,cp,rp,sato,table}.py – fine‐tuning heads

F. Training & Experiment Management (purple)
- CLI entrypoints:
– pretrain: scripts/pretrain.py
– fine‐tuning: train.py
- exp/…/*.jsonnet + *.yml – experiment configurations (hyperparameters, schedules)
- Uses Apex for mixed‐precision

G. Model Artifacts & Output Storage (brown)
- out/ – output model directory (out_model/)
- model/ – pretrained archives
- logs, predictions, saved weights

H. Prediction Pipeline (teal)
- pretrain_pred.py – corrupt‐cell detection inference
- pred.py – fine‐tuned task inference
- table_embedder/predictors – predictor classes (cell_predictor, generic predictor)

##### Relationships & Data Flow
1. Raw Data → Preprocessing Scripts → Preprocessed Files (npy, token IDs)
2. Preprocessed Files → Data Readers → Torch Datasets & DataLoaders
3. Datasets → Model Library → Training Loop (train.py / scripts/pretrain.py)
4. Training Loop ↔ Experiment Config (exp/*.yml)
5. Training Loop → Model Artifacts (out_model/)
6. Model Artifacts + Test Data → Prediction Scripts → Final Predictions

##### Architectural Patterns & Design Principles
- Modular “reader → model → predictor” pipeline for each task
- Separation of concerns:
- scripts/ → orchestration & utilities
- table_embedder/ → core library (IO, model, prediction)
- exp/ → configuration management
- Use of config‐as‐code (jsonnet + YAML) for reproducibility
- Plugin‐style readers & predictors to handle different tasks

##### Elements
- Data Sources
- Preprocessing & Feature Extraction
- Data Readers
- Model Library
- Training & Experiment Manager
- Model Artifacts
- Prediction Pipeline
- Call out config files & environment dependencies on the side

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        DS1["Pretraining JSONL"]:::data
        DS2["FT Cell Data Dir"]:::data
        DS3["FT Col Data Dir"]:::data
        DS4["FT Table Data Dir"]:::data
        DS5["BERT Vocab"]:::data
    end

    %% Environment & Dependencies
    subgraph "Environment & Dependencies"
        ENV["Conda Env"]:::env
        PM1["freq.tar.gz"]:::env
        PM2["mix.tar.gz"]:::env
    end

    %% Preprocessing & Feature Extraction
    subgraph "Preprocessing & Feature Extraction"
        P1["scripts/to_npy.py"]:::preproc
        P2["ext_feat.py"]:::preproc
        P3["scripts/bert.py"]:::preproc
        P4["scripts/col_pop.py"]:::preproc
        P5["scripts/row_pop.py"]:::preproc
        P6["scripts/sato.py"]:::preproc
    end

    %% Data Readers
    subgraph "Data Readers"
        DR1["cell_reader.py"]:::reader
        DR2["cp_reader.py"]:::reader
        DR3["rp_reader.py"]:::reader
        DR4["sato_reader.py"]:::reader
        DR5["pretrain_reader.py"]:::reader
        DR6["write_callback.py"]:::reader
    end

    %% Model Library
    subgraph "Model Library"
        subgraph "Core Building Blocks"
            ML1["bert_token_embedder.py"]:::model
            ML2["multi_head_self_attention.py"]:::model
            ML3["stacked_self_attention.py"]:::model
            ML4["masked_ff.py"]:::model
        end
        ML5["pretrain.py"]:::model
        subgraph "Fine-tuning Heads"
            ML6["finetune_cell.py"]:::model
            ML7["finetune_col.py"]:::model
            ML8["finetune_cp.py"]:::model
            ML9["finetune_rp.py"]:::model
            ML10["finetune_sato.py"]:::model
            ML11["finetune_table.py"]:::model
        end
    end

    %% Training & Experiment Management
    subgraph "Training & Experiment Management"
        T1["scripts/pretrain.py"]:::train
        T2["train.py"]:::train
        subgraph "Experiment Configs"
            C1["exp/pretrain/"]:::train
            C2["exp/ft_cell/"]:::train
            C3["exp/ft_col/"]:::train
            C4["exp/ft_table/"]:::train
            C5["exp/sato/"]:::train
        end
    end

    %% Model Artifacts & Output Storage
    subgraph "Model Artifacts & Output Storage"
        O1["out/"]:::artifacts
        O2["model/"]:::artifacts
    end

    %% Prediction Pipeline
    subgraph "Prediction Pipeline"
        PR1["pretrain_pred.py"]:::pred
        PR2["pred.py"]:::pred
        subgraph "Predictors"
            PR3["cell_predictor.py"]:::pred
            PR4["predictor.py"]:::pred
        end
    end

    %% Data Flow
    DS1 -->|provides JSONL| P1
    DS2 -->|source CSVs| P1
    DS3 -->|source CSVs| P1
    DS4 -->|source CSVs| P1
    DS5 -->|vocab| P3

    P1 -->|npy arrays| DR1
    P1 -->|npy arrays| DR2
    P1 -->|npy arrays| DR3
    P1 -->|npy arrays| DR4
    P1 -->|npy arrays| DR5
    P2 --> DR* 
    P3 --> DR* 
    P4 --> DR* 
    P5 --> DR* 
    P6 --> DR* 

    DR1 -->|"Torch Dataset"| T1
    DR2 -->|"Torch Dataset"| T1
    DR3 -->|"Torch Dataset"| T1
    DR4 -->|"Torch Dataset"| T1
    DR5 -->|"Torch Dataset"| T1

    T1 -->|uses Models| ML5
    T1 -->|uses Core| ML1
    T1 -->|uses Heads| ML6
    T1 -->|"writes Artifacts"| O1

    T2 -->|uses Models| ML6
    T2 -->|uses Core| ML2
    T2 -->|uses Heads| ML7
    T2 -->|"writes Artifacts"| O1

    C1 --> T1
    C2 --> T2
    C3 --> T2
    C4 --> T2
    C5 --> T2

    O1 -->|trained models| PR1
    O1 -->|trained models| PR2
    DS2 --> PR2
    DS3 --> PR2
    DS4 --> PR2

    PR1 --> PR3
    PR2 --> PR4

    ENV --> P1
    ENV --> T1
    ENV --> T2

    PM1 --> ENV
    PM2 --> ENV

    %% Click Events
    click DS1 "https://github.com/sfig611/tabbie/blob/main/data/pretrain/sample.jsonl"
    click DS2 "https://github.com/sfig611/tabbie/tree/main/data/ft_cell/"
    click DS3 "https://github.com/sfig611/tabbie/tree/main/data/ft_col/"
    click DS4 "https://github.com/sfig611/tabbie/tree/main/data/ft_table/"
    click DS5 "https://github.com/sfig611/tabbie/blob/main/data/bert-base-uncased-vocab.txt"
    click ENV "https://github.com/sfig611/tabbie/blob/main/env/env.yml"
    click PM1 "https://github.com/sfig611/tabbie/blob/main/model/freq.tar.gz"
    click PM2 "https://github.com/sfig611/tabbie/blob/main/model/mix.tar.gz"
    click P1 "https://github.com/sfig611/tabbie/blob/main/scripts/to_npy.py"
    click P2 "https://github.com/sfig611/tabbie/blob/main/ext_feat.py"
    click P3 "https://github.com/sfig611/tabbie/blob/main/scripts/bert.py"
    click P4 "https://github.com/sfig611/tabbie/blob/main/scripts/col_pop.py"
    click P5 "https://github.com/sfig611/tabbie/blob/main/scripts/row_pop.py"
    click P6 "https://github.com/sfig611/tabbie/blob/main/scripts/sato.py"
    click DR1 "https://github.com/sfig611/tabbie/blob/main/table_embedder/readers/cell_reader.py"
    click DR2 "https://github.com/sfig611/tabbie/blob/main/table_embedder/readers/cp_reader.py"
    click DR3 "https://github.com/sfig611/tabbie/blob/main/table_embedder/readers/rp_reader.py"
    click DR4 "https://github.com/sfig611/tabbie/blob/main/table_embedder/readers/sato_reader.py"
    click DR5 "https://github.com/sfig611/tabbie/blob/main/table_embedder/readers/pretrain_reader.py"
    click DR6 "https://github.com/sfig611/tabbie/blob/main/table_embedder/readers/epoch_callbacks/write_callback.py"
    click ML1 "https://github.com/sfig611/tabbie/blob/main/table_embedder/models/lib/bert_token_embedder.py"
    click ML2 "https://github.com/sfig611/tabbie/blob/main/table_embedder/models/lib/multi_head_self_attention.py"
    click ML3 "https://github.com/sfig611/tabbie/blob/main/table_embedder/models/lib/stacked_self_attention.py"
    click ML4 "https://github.com/sfig611/tabbie/blob/main/table_embedder/models/lib/masked_ff.py"
    click ML5 "https://github.com/sfig611/tabbie/blob/main/table_embedder/models/pretrain.py"
    click ML6 "https://github.com/sfig611/tabbie/blob/main/table_embedder/models/finetune_cell.py"
    click ML7 "https://github.com/sfig611/tabbie/blob/main/table_embedder/models/finetune_col.py"
    click ML8 "https://github.com/sfig611/tabbie/blob/main/table_embedder/models/finetune_cp.py"
    click ML9 "https://github.com/sfig611/tabbie/blob/main/table_embedder/models/finetune_rp.py"
    click ML10 "https://github.com/sfig611/tabbie/blob/main/table_embedder/models/finetune_sato.py"
    click ML11 "https://github.com/sfig611/tabbie/blob/main/table_embedder/models/finetune_table.py"
    click T1 "https://github.com/sfig611/tabbie/blob/main/scripts/pretrain.py"
    click T2 "https://github.com/sfig611/tabbie/blob/main/train.py"
    click C1 "https://github.com/sfig611/tabbie/tree/main/exp/pretrain/"
    click C2 "https://github.com/sfig611/tabbie/tree/main/exp/ft_cell/"
    click C3 "https://github.com/sfig611/tabbie/tree/main/exp/ft_col/"
    click C4 "https://github.com/sfig611/tabbie/tree/main/exp/ft_table/"
    click C5 "https://github.com/sfig611/tabbie/tree/main/exp/sato/"
    click O1 "https://github.com/sfig611/tabbie/tree/main/out/"
    click O2 "https://github.com/sfig611/tabbie/tree/main/model/"
    click PR1 "https://github.com/sfig611/tabbie/blob/main/pretrain_pred.py"
    click PR2 "https://github.com/sfig611/tabbie/blob/main/pred.py"
    click PR3 "https://github.com/sfig611/tabbie/blob/main/table_embedder/predictors/cell_predictor.py"
    click PR4 "https://github.com/sfig611/tabbie/blob/main/table_embedder/predictors/predictor.py"

    %% Styles
    classDef data fill:#CFE2F3,stroke:#000,stroke-width:1px
    classDef env fill:#E5E5E5,stroke:#000,stroke-width:1px
    classDef preproc fill:#D9EAD3,stroke:#000,stroke-width:1px
    classDef reader fill:#FFF2CC,stroke:#000,stroke-width:1px
    classDef model fill:#F9CB9C,stroke:#000,stroke-width:1px
    classDef train fill:#D5A6BD,stroke:#000,stroke-width:1px
    classDef artifacts fill:#D9D2E9,stroke:#000,stroke-width:1px
    classDef pred fill:#B5EAD7,stroke:#000,stroke-width:1px
```
