How sdv-dev/ctgan works
Project type and purpose
• This is a Python library (ctgan/) implementing two deep-learning synthetic data generators (CTGAN & TVAE) plus associated preprocessing.
• It exposes both a programmatic API and a CLI entrypoint (ctgan/__main__.py).
• It bundles sample data loaders, data transformers, and synthesizer implementations as core components.
File structure
• ctgan/ (core package)
• __main__.py – CLI entrypoint
• data.py – data‐loading logic (load_demo)
• data_transformer.py – transforms between raw & model‐ready formats
• data_sampler.py – sampling utilities and conditional vector logic
• synthesizers/ – base classes and concrete synthesizer implementations (ctgan.py, tvae.py)
• demo.py – wrapper combining load → fit → sample flow
• errors.py – custom exception types
• examples/ – sample CSV/TSV datasets & metadata to demonstrate library usage
• tests/ – unit & integration tests organized by component (data transformer, synthesizers)
• scripts/ & tasks.py – automation (release notes, Makefile targets)
• CI/CD configuration under .github/workflows and tox.ini
Extract additional insights from the README
• CTGAN implements the models described in the “Conditional GAN” NeurIPS paper.
• Primary user flows:
a. load_demo() → returns real DataFrame
b. CTGAN.fit(real, discrete_cols) → trains model
c. CTGAN.sample(n) → returns synthetic DataFrame
• The library can be invoked standalone or via the broader SDV ecosystem.
Main components & interactions
a. User / Consumer
• Python script or interactive REPL
• CLI (ctgan console script)
b. Data Loader (ctgan.data)
• Fetches raw demo data or user‐provided data
c. Data Transformer (ctgan.data_transformer)
• Converts pandas DataFrame → numerical tensors, one-hot, normalization
• Stores transformer state for inverse mapping
d. Conditional Vector & Sampler (ctgan.data_sampler)
• Computes conditional vectors for discrete columns
• Samples conditioning vectors at generate time
e. Synthesizer Core (ctgan.synthesizers.base)
• Defines fit(), sample() interface
f. Model Implementations
• CTGAN (ctgan.synthesizers.ctgan) – conditional GAN architecture
• TVAE (ctgan.synthesizers.tvae) – variational autoencoder variant
g. Output Transformer (inverse in data_transformer)
• Maps model output back → pandas DataFrame with original dtypes
h. CLI / Demo (ctgan/__main__.py, ctgan/demo.py)
• Glue code orchestrating above steps end-to-end
i. Tests & CI
• Unit tests for each module
• Integration tests for full fit→sample pipeline
• GitHub Actions workflows for lint, unit, integration, release
Architectural patterns & design principles
• Modular Package Layout: clear separation of concerns (data, transform, sampler, models).
• Strategy Pattern: base synthesizer class + pluggable model implementations.
• ETL Pipeline: data ingestion → transform → model training → sample → inverse transform.
• CLI + API Dual‐Interface: same core logic exposed programmatically and via console.
• Test‐Driven: comprehensive unit/integration test suite safeguarding each component.
• Technologies: Python, PyTorch (implied by CTGAN paper), pandas.