How minkaixu/tabdiff works
Project Type and Purpose
This is a research-oriented machine learning framework for generative modeling of tabular data. It provides:
– Data ingestion and preprocessing pipelines
– A unified continuous‐time diffusion model (TabDiff) implementation
– Training and sampling command‐line interfaces (CLIs)
– Evaluation modules for quality, privacy, and imputation metrics
High-Level Components
a. Data Preparation
• download_dataset.py: fetches raw CSVs
• process_dataset.py: converts raw CSV into standardized format and metadata JSON under data/ and data/Info/
b. Core Module (tabdiff/)
• configs: model and experiment parameters (tabdiff_configs.toml)
• models: diffusion model definitions (unified C-time, noise schedules)
• modules: network building blocks (transformers, main_modules)
• trainer.py: training loop orchestration
• main.py: entry point parsing CLI args, invoking training or sampling
• metrics.py: specialized metrics used during training or sampling
c. Orchestration Scripts
• main.py (root): high‐level CLI for train/test/report/impute
• utils_train.py: helper routines for logging, checkpointing, environment setups
• src/: supporting utilities (data loaders, environment config, metric wrappers)
d. Evaluation Suite (eval/)
• eval_quality.py: computes α-precision/β-recall
• mle/: maximum likelihood estimation module
• visualize_density.py: density‐distribution comparisons
• eval_impute.py: imputation performance
e. Environments and Configs
• tabdiff.yaml: conda environment for core functionality
• synthcity.yaml: conda environment for additional metrics
f. Artifacts
• data/: raw and processed datasets and metadata
• eval/report_runs/: saved evaluation results and logs
• impute/: saved imputation outputs
• images/: demo videos and flowcharts
Relationships and Data Flow
1. start → download_dataset.py → raw CSVs in data/
2. process_dataset.py → standardized data + Info/*.json
3. root main.py (--mode train) → env setup → tabdiff/main.py → trainer.py → tabdiff/models & modules → checkpoints
4. root main.py (--mode test) → loads checkpoints → sampling via unified_ctime_diffusion → synthetic tables
5. test with --report → invokes src/metrics.py → writes to eval/report_runs
6. eval/eval_quality.py (in synthcity env) → reads samples → computes α/β scores
7. For imputation (--impute) → unified_ctime_diffusion.sample_impute → impute/
8. eval_impute.py → computes imputation accuracy
Architectural Patterns and Principles
• Modular design separating core ML logic (tabdiff/) from orchestration (root scripts) and utilities (src/)
• CLI-driven workflow: root main.py dispatches to appropriate modules
• Config-driven experiments: toml and yaml for reproducibility
• Pluggable evaluation: independent eval/ scripts that consume outputs
• Clear separation of concerns: data, model, training, sampling, evaluation