# zhaohualu/syn architecture

> Architecture diagram of the GitHub repository zhaohualu/syn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zhaohualu/syn
- Repository: https://github.com/zhaohualu/syn

## Overview

##### Project Overview and Primary Purpose
- This repository is a research-oriented codebase aimed at “Boosting Data Analytics with Synthetic Volume Expansion.” It contains multiple experiment pipelines (e.g., Syn-Slm, Syn-Boost, Syn-Test) that work with synthetic data generation and analysis using conditional generative models as well as tabular diffusion techniques.
- The project covers several use cases:
– Sentiment analysis (with models such as DistilBERT and LSTM).
– Regression simulation tasks on tabular data using conditional generative methods.
– Prediction and inference experiments where synthetic data is tuned for optimal predictive performance (using CatBoost, optuna tuning, etc.).

##### Analyzing the File Structure
- Top-Level Project Organization:
– The files are grouped into three main directories: “conditional”, “sentiment”, and “tab-ddpm”.
- “conditional”: Houses scripts and utilities (e.g., ablation_sigma.py) for running regression experiments on tabular data with different sigma parameters.
- “sentiment”: Contains implementations and notebooks for conducting sentiment analysis, including evaluation using GPT-3.5 (via gpt_result.csv) and fine-tuning approaches.
- “tab-ddpm”: This is the largest section and is subdivided into modules such as “lib”, “scripts”, “syninf”, “synpred”, and “tab_ddpm”. These parts include:
▪ Definition of the tabular diffusion model and common utilities.
▪ Training, evaluation, resampling, and tuning scripts.
▪ Directories for simulation inference (“syninf”) and prediction boosting (“synpred”), which contain notebooks, checkpoints, and optuna studies.
▪ “utils” providing additional functions for visualization and synthetic data processing.
- Supporting files such as environment.yml, poetry/pyproject, and various requirements files establish the dependencies and build configurations.

A. Identify Major Components:
– Environment & Dependency Management: Represent the conda and poetry setup that ensures all libraries (e.g., pytorch, diffusion model dependencies) are installed.
– Data Ingestion & Preprocessing: Show separate data directories under “conditional”, “sentiment”, and “tab-ddpm” that feed raw/tabular data or text data.
– Model Training Components:
- For tabular regression (“conditional”) – Scripts like ablation_sigma.py run experiments with different model parameters.
- For diffusion-based generative modeling (“tab-ddpm/tab_ddpm” and “lib”) – Core diffusion model modules (like gaussian_multinomial_diffsuion.py) supported by training pipelines in “scripts/train.py” and tuning scripts.
- For sentiment analysis (“sentiment”) – Implements both transformer-based and LSTM approaches and handles checkpointing.
– Inference & Evaluation:
- Prediction Boost (in synpred): Displays how synthetic data is used alongside raw data. Include the optuna tuning modules (ratio_optuna_studies) and evaluation notebooks.
- Synthetic Test (in syninf): Represents inference on actual datasets (e.g., adult, california) through notebooks and the evaluation pipeline.
– Results & Metrics Storage: Include “result” directories where outputs (csv, pickle files, loss metrics, etc.) are saved.
– External Interactions: Indicate the use of external services like GPT-3.5 and external pre-trained model checkpoints (for DistilBERT in sentiment).

B. Representing Components and Their Relationships:
– Use distinct blocks for each major component area:
◦ Environment Setup (Conda, Poetry) at the top or side as a foundational layer.
◦ Data Sources/Preprocessing block feeding into the model training areas.
▪ For example, data flows from raw data storage to preprocessing scripts, then to training pipelines, and finally to evaluation.
– Label connections that indicate dependencies (e.g., “pre-trained checkpoints”, “evaluation notebooks”).

◦ Rectangles for data sources and storage directories.
◦ Diamonds for decision points (like tuning with optuna).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Environment & Dependency Management
    subgraph "Environment & Dependency Management"
        ENV1["environment.yml"]:::env
        ENV2["pyproject.toml"]:::env
        ENV3["requirements.txt"]:::env
        ENV4["requirement.txt"]:::env
        ENV5["poetry2pip.py"]:::env
    end

    %% Data Ingestion & Preprocessing
    subgraph "Data Ingestion & Preprocessing"
        DAT_Cond["conditional/data"]:::data
        DAT_Sent["sentiment/data"]:::data
        DAT_Tab["tab-ddpm/lib/data"]:::data
    end

    %% Connect Environment to Data Ingestion (using representative nodes)
    ENV1 -->|"provides"| DAT_Cond
    ENV2 -->|"provides"| DAT_Sent
    ENV3 -->|"provides"| DAT_Tab

    %% Tabular Regression (Conditional Model Training)
    subgraph "Tabular Regression (Conditional Model Training)"
        CR1["conditional/ablation_sigma.py"]:::training
        CR2["conditional/utils"]:::training
        CR_RES["conditional/result"]:::evaluation
    end

    %% Data flow for Regression Experiments
    DAT_Cond -->|"input_data"| CR1
    CR1 -->|"experiment_output"| CR_RES

    %% Sentiment Analysis Model Training
    subgraph "Sentiment Analysis Model Training"
        SA1["sentiment/imdb_distillbert.py"]:::training
        SA2["sentiment/imdb_lstm.py"]:::training
        SA3["sentiment/imdb_lstm.ipynb"]:::training
        SA_RES["sentiment/result/gpt_result.csv"]:::evaluation
        SA_EXT["GPT-3.5_API"]:::external
    end

    %% Data flow for Sentiment Analysis
    DAT_Sent -->|"input_data"| SA1
    SA1 -->|"next_step"| SA2
    SA2 -->|"next_step"| SA3
    SA3 -->|"evaluation_output"| SA_RES
    SA_EXT -->|"external_eval"| SA_RES

    %% Tabular Diffusion (Synpred & Syninf)
    subgraph "Tabular Diffusion (Synpred & Syninf)"
        %% Diffusion Model Training Subgraph
        subgraph "Diffusion Model Training"
            TD1["tab-ddpm/tab_ddpm/gaussian_multinomial_diffsuion.py"]:::training
            TD2["tab-ddpm/scripts/train.py"]:::training
            TD3["tab-ddpm/tab_ddpm/modules.py"]:::training
            TD4["tab-ddpm/tab_ddpm/utils_misc.py"]:::training
            TD_LIB["tab-ddpm/lib"]:::training
        end
        %% Inference & Prediction Subgraph
        subgraph "Inference & Prediction"
            DI1["tab-ddpm/syninf"]:::evaluation
            DI2["tab-ddpm/synpred"]:::evaluation
            DI3["tab-ddpm/synpred/ratio_optuna_studies"]:::evaluation
            DI_RES["Diffusion_Results"]:::evaluation
        end
    end

    %% Data flow for Diffusion Experiments
    DAT_Tab -->|"raw_tabular_data"| TD1
    TD1 -->|"training"| TD2
    TD2 -->|"trained_model"| DI1
    TD2 -->|"trained_model"| DI2
    DI2 -->|"tuning"| DI3
    DI3 -->|"results"| DI_RES
    DI1 -->|"inference_output"| DI_RES

    %% Click Events for Environment & Dependency Management
    click ENV1 "https://github.com/zhaohualu/syn/blob/main/environment.yml"
    click ENV2 "https://github.com/zhaohualu/syn/blob/main/pyproject.toml"
    click ENV3 "https://github.com/zhaohualu/syn/blob/main/requirements.txt"
    click ENV4 "https://github.com/zhaohualu/syn/blob/main/requirement.txt"
    click ENV5 "https://github.com/zhaohualu/syn/blob/main/poetry2pip.py"

    %% Click Events for Data Ingestion & Preprocessing
    click DAT_Cond "https://github.com/zhaohualu/syn/tree/main/conditional/data"
    click DAT_Sent "https://github.com/zhaohualu/syn/tree/main/sentiment/data"
    click DAT_Tab "https://github.com/zhaohualu/syn/tree/main/tab-ddpm/lib/data"

    %% Click Events for Tabular Regression
    click CR1 "https://github.com/zhaohualu/syn/blob/main/conditional/ablation_sigma.py"
    click CR2 "https://github.com/zhaohualu/syn/tree/main/conditional/utils"
    click CR_RES "https://github.com/zhaohualu/syn/tree/main/conditional/result"

    %% Click Events for Sentiment Analysis
    click SA1 "https://github.com/zhaohualu/syn/blob/main/sentiment/imdb_distillbert.py"
    click SA2 "https://github.com/zhaohualu/syn/blob/main/sentiment/imdb_lstm.py"
    click SA3 "https://github.com/zhaohualu/syn/blob/main/sentiment/imdb_lstm.ipynb"
    click SA_RES "https://github.com/zhaohualu/syn/blob/main/sentiment/result/gpt_result.csv"

    %% Click Events for Diffusion-based Generative Modeling
    click TD1 "https://github.com/zhaohualu/syn/blob/main/tab-ddpm/tab_ddpm/gaussian_multinomial_diffsuion.py"
    click TD2 "https://github.com/zhaohualu/syn/blob/main/tab-ddpm/scripts/train.py"
    click TD3 "https://github.com/zhaohualu/syn/blob/main/tab-ddpm/tab_ddpm/modules.py"
    click TD4 "https://github.com/zhaohualu/syn/blob/main/tab-ddpm/tab_ddpm/utils_misc.py"
    click TD_LIB "https://github.com/zhaohualu/syn/tree/main/tab-ddpm/lib"

    %% Click Events for Inference & Evaluation in Diffusion
    click DI1 "https://github.com/zhaohualu/syn/tree/main/tab-ddpm/syninf"
    click DI2 "https://github.com/zhaohualu/syn/tree/main/tab-ddpm/synpred"
    click DI3 "https://github.com/zhaohualu/syn/tree/main/tab-ddpm/synpred/ratio_optuna_studies"

    %% Styles
    classDef env fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef data fill:#98FB98,stroke:#000,stroke-width:2px;
    classDef training fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef evaluation fill:#FFA500,stroke:#000,stroke-width:2px;
    classDef external fill:#FFB6C1,stroke:#000,stroke-width:2px;
```
