# minkaixu/tabdiff architecture

> Architecture diagram of the GitHub repository minkaixu/tabdiff, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/minkaixu/tabdiff
- Repository: https://github.com/minkaixu/tabdiff

## Overview

##### Project Type and Purpose
This is a research-oriented machine learning framework for generative modeling of tabular data. It provides:
– Data ingestion and preprocessing pipelines
– A unified continuous‐time diffusion model (TabDiff) implementation
– Training and sampling command‐line interfaces (CLIs)
– Evaluation modules for quality, privacy, and imputation metrics

##### High-Level Components
a. Data Preparation
- download_dataset.py: fetches raw CSVs
- process_dataset.py: converts raw CSV into standardized format and metadata JSON under data/ and data/Info/
b. Core Module (tabdiff/)
- configs: model and experiment parameters (tabdiff_configs.toml)
- models: diffusion model definitions (unified C-time, noise schedules)
- modules: network building blocks (transformers, main_modules)
- trainer.py: training loop orchestration
- main.py: entry point parsing CLI args, invoking training or sampling
- metrics.py: specialized metrics used during training or sampling
c. Orchestration Scripts
- main.py (root): high‐level CLI for train/test/report/impute
- utils_train.py: helper routines for logging, checkpointing, environment setups
- src/: supporting utilities (data loaders, environment config, metric wrappers)
d. Evaluation Suite (eval/)
- eval_quality.py: computes α-precision/β-recall
- mle/: maximum likelihood estimation module
- visualize_density.py: density‐distribution comparisons
- eval_impute.py: imputation performance
e. Environments and Configs
- tabdiff.yaml: conda environment for core functionality
- synthcity.yaml: conda environment for additional metrics
f. Artifacts
- data/: raw and processed datasets and metadata
- eval/report_runs/: saved evaluation results and logs
- impute/: saved imputation outputs
- images/: demo videos and flowcharts

##### Relationships and Data Flow
1. start → download_dataset.py → raw CSVs in data/
2. process_dataset.py → standardized data + Info/*.json
3. root main.py (--mode train) → env setup → tabdiff/main.py → trainer.py → tabdiff/models & modules → checkpoints
4. root main.py (--mode test) → loads checkpoints → sampling via unified_ctime_diffusion → synthetic tables
5. test with --report → invokes src/metrics.py → writes to eval/report_runs
6. eval/eval_quality.py (in synthcity env) → reads samples → computes α/β scores
7. For imputation (--impute) → unified_ctime_diffusion.sample_impute → impute/
8. eval_impute.py → computes imputation accuracy

##### Architectural Patterns and Principles
- Modular design separating core ML logic (tabdiff/) from orchestration (root scripts) and utilities (src/)
- CLI-driven workflow: root main.py dispatches to appropriate modules
- Config-driven experiments: toml and yaml for reproducibility
- Pluggable evaluation: independent eval/ scripts that consume outputs
- Clear separation of concerns: data, model, training, sampling, evaluation

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Pipeline (Blue)
    subgraph "Data Pipeline"
        classDef data fill:#D0E6FF,stroke:#7AAEFF
        DL["Download Dataset"]:::data
        PD["Process Dataset"]:::data
        RAW[(Raw CSVs)]:::data
        PROCESSED[(Processed Data)]:::data
        INFO[(Metadata JSONs)]:::data
    end

    DL -->|download| RAW
    RAW -->|standardize| PD
    PD --> PROCESSED
    PD --> INFO

    click DL "https://github.com/minkaixu/tabdiff/blob/main/download_dataset.py"
    click PD "https://github.com/minkaixu/tabdiff/blob/main/process_dataset.py"

    %% CLI Orchestrator (Gray)
    subgraph "CLI Orchestrator"
        classDef orchestrator fill:#F0F0F0,stroke:#888888
        ROOT["Root CLI (main.py)"]:::orchestrator
        UTILS["Train Utils"]:::orchestrator
        subgraph "Supporting Utilities"
            SRC_DATA["src/data.py"]:::orchestrator
            SRC_ENV["src/env.py"]:::orchestrator
            SRC_METRICS["src/metrics.py"]:::orchestrator
            SRC_UTIL["src/util.py"]:::orchestrator
        end
    end

    ROOT --> UTILS
    ROOT -->|dispatch| TAB_MAIN

    click ROOT "https://github.com/minkaixu/tabdiff/blob/main/main.py"
    click UTILS "https://github.com/minkaixu/tabdiff/blob/main/utils_train.py"
    click SRC_DATA "https://github.com/minkaixu/tabdiff/blob/main/src/data.py"
    click SRC_ENV "https://github.com/minkaixu/tabdiff/blob/main/src/env.py"
    click SRC_METRICS "https://github.com/minkaixu/tabdiff/blob/main/src/metrics.py"
    click SRC_UTIL "https://github.com/minkaixu/tabdiff/blob/main/src/util.py"

    %% TabDiff Core (Green)
    subgraph "TabDiff Core"
        classDef core fill:#DFF8E2,stroke:#3ECB3E
        CFG["Configs (toml)"]:::core
        TAB_MAIN["tabdiff/main.py"]:::core
        TRAINER["Trainer"]:::core
        subgraph "Models"
            MODEL1["unified_ctime_diffusion.py"]:::core
            MODEL2["noise_schedule.py"]:::core
        end
        subgraph "Modules"
            MOD1["main_modules.py"]:::core
            MOD2["transformer.py"]:::core
        end
        MET_CORE["tabdiff/metrics.py"]:::core
        CKPT[(Checkpoints)]:::core
    end

    CFG --> TAB_MAIN
    CFG --> TRAINER
    TAB_MAIN -->|train| TRAINER
    TRAINER --> MODEL1
    TRAINER --> MODEL2
    TRAINER --> MOD1
    TRAINER --> MOD2
    TRAINER --> MET_CORE
    TRAINER --> CKPT
    TAB_MAIN -->|sample| SYNTH

    click CFG "https://github.com/minkaixu/tabdiff/blob/main/tabdiff/configs/tabdiff_configs.toml"
    click TAB_MAIN "https://github.com/minkaixu/tabdiff/blob/main/tabdiff/main.py"
    click TRAINER "https://github.com/minkaixu/tabdiff/blob/main/tabdiff/trainer.py"
    click MODEL1 "https://github.com/minkaixu/tabdiff/blob/main/tabdiff/models/unified_ctime_diffusion.py"
    click MODEL2 "https://github.com/minkaixu/tabdiff/blob/main/tabdiff/models/noise_schedule.py"
    click MOD1 "https://github.com/minkaixu/tabdiff/blob/main/tabdiff/modules/main_modules.py"
    click MOD2 "https://github.com/minkaixu/tabdiff/blob/main/tabdiff/modules/transformer.py"
    click MET_CORE "https://github.com/minkaixu/tabdiff/blob/main/tabdiff/metrics.py"

    %% Synthetic Data Store
    SYNTH[(Synthetic Tables)]:::data

    %% Imputation Outputs
    IMP[(Imputation Outputs)]:::data
    TAB_MAIN -->|impute| IMP

    %% Evaluation Suite (Orange)
    subgraph "Evaluation Suite"
        classDef eval fill:#FFE6C0,stroke:#FFAE42
        EQ["eval_quality.py"]:::eval
        subgraph "MLE Module"
            MLE1["mle.py"]:::eval
            MLE2["tabular_dataload.py"]:::eval
            MLE3["tabular_transformer.py"]:::eval
        end
        VD["visualize_density.py"]:::eval
        EI["eval_impute.py"]:::eval
        REPORT[(Report Runs)]:::eval
    end

    SYNTH -->|compute α/β| EQ
    SYNTH -->|compute MLE| MLE1
    MLE1 --> MLE2
    MLE1 --> MLE3
    SYNTH -->|visualize| VD
    IMP -->|evaluate impute| EI
    EQ --> REPORT
    MLE1 --> REPORT
    VD --> REPORT
    EI --> REPORT

    click EQ "https://github.com/minkaixu/tabdiff/blob/main/eval/eval_quality.py"
    click MLE1 "https://github.com/minkaixu/tabdiff/blob/main/eval/mle/mle.py"
    click MLE2 "https://github.com/minkaixu/tabdiff/blob/main/eval/mle/tabular_dataload.py"
    click MLE3 "https://github.com/minkaixu/tabdiff/blob/main/eval/mle/tabular_transformer.py"
    click VD "https://github.com/minkaixu/tabdiff/blob/main/eval/visualize_density.py"
    click EI "https://github.com/minkaixu/tabdiff/blob/main/eval_impute.py"

    %% Environments (Gray)
    subgraph "Environments"
        classDef cfg fill:#E0E0E0,stroke:#888888
        ENV1["tabdiff.yaml"]:::cfg
        ENV2["synthcity.yaml"]:::cfg
    end

    ENV1 -.-> TRAINER
    ENV2 -.-> EQ

    click ENV1 "https://github.com/minkaixu/tabdiff/blob/main/tabdiff.yaml"
    click ENV2 "https://github.com/minkaixu/tabdiff/blob/main/synthcity.yaml"

    %% Artifacts
    subgraph "Artifacts"
        classDef artifacts fill:#F8F8F8,stroke:#AAAAAA
        IMG["Demo Video"]:::artifacts
    end

    click IMG "https://github.com/minkaixu/tabdiff/blob/main/images/tabdiff_demo.mp4"

    %% Legend
    classDef legendTitle fill:#ffffff,stroke:#000000,font-weight:bold
    classDef dataPipeline,data core,eval,cfg,orchestrator,artifacts fill-opacity:0.2
    subgraph Legend
        direction TB
        L1["■ Data Pipeline (Blue)"]:::data
        L2["■ TabDiff Core (Green)"]:::core
        L3["■ Evaluation Suite (Orange)"]:::eval
        L4["■ Environments (Gray)"]:::cfg
        L5["■ Orchestration (Gray)"]:::orchestrator
        L6["■ Artifacts (Light Gray)"]:::artifacts
    end
```
