# arashdn/tabulax architecture

> Architecture diagram of the GitHub repository arashdn/tabulax, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arashdn/tabulax
- Repository: https://github.com/arashdn/tabulax

## Overview

##### Project type and purpose
- This is a research‐oriented, command‐line data‐processing pipeline (“TabulaX”) that automates multi‐class table transformations using large language models (LLMs).
- It ingests labeled CSV datasets, classifies each table’s transformation type (e.g., algorithmic, string, numeric, general), invokes an LLM (either OpenAI’s GPT family or a local LLaMA 3 served via vLLM) to generate the transformed output, and evaluates results against ground truth.

##### File structure
- Top‐level directories:
- data/ — contains three datasets (AutoJoin, FlashFill, DataXFormer) organized by table, each with source.csv, target.csv, ground truth.
- src/LLM_pipeline/ — the code under three modules:
– classifier/ — input classifier scripts (gpt_classifier.py, llama_classifier.py), prompt templates, utilities, metrics reporter
– transformers/ — transformation functions grouped by class (basic, algorithmic, general, numeric, string) with their prompt templates
– util/ — dataset loader/sampler and join‐evaluation utilities
- Orchestration script: run_pipeline.py invokes classification and transformation in sequence.
- Config files: environment.yml, requirements.txt, plus several cache‐path constants in the code (PROMPT_CACHE_PATH, ED_CACHE_PATH).

##### Extract architectural clues from the README
- Two phases: classification of input tables, then transformation of each table.
- Dual model support: OpenAI API vs. local LLaMA via vLLM.
- Caching layers for prompt generation and edit‐distance computations.
- Pluggable “matching” and “classification” strategies configured at runtime.

##### Essential details
- Run_pipeline.py as the orchestrator at the center.
- Configuration inputs (MODEL_NAME, EXAMPLE_SIZE, MATCHING_TYPE, CLASSIFICATION_TYPE) feeding into the orchestrator.
- Detail prompt paths (classifier_prompts/*, transformers/prompts/*) linking into classifier and transformer modules.
- Evaluation outputs: performance_report.csv, metrics printed by report_metrics.py and JoinEval.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration
    Config["MODEL_NAME, CLASSIFICATION_TYPE, MATCHING_TYPE, EXAMPLE_SIZE"]:::config
    click Config "https://github.com/arashdn/tabulax/blob/main/environment.yml"
    click Config "https://github.com/arashdn/tabulax/blob/main/requirements.txt"

    %% Orchestrator
    RP["run_pipeline.py"]:::core
    click RP "https://github.com/arashdn/tabulax/blob/main/src/LLM_pipeline/run_pipeline.py"

    Config --> RP

    %% Data Layer
    subgraph "Data Layer"
        Data["Datasets (source.csv, target.csv, ground_truth)"]:::data
    end
    click Data "https://github.com/arashdn/tabulax/tree/main/data/Datasets"
    RP --> Data

    %% Preprocessor
    subgraph "Preprocessor"
        Preproc["preprocessors.py"]:::core
        Loader["util/dataset.py"]:::core
    end
    click Preproc "https://github.com/arashdn/tabulax/blob/main/src/LLM_pipeline/preprocessors.py"
    click Loader "https://github.com/arashdn/tabulax/blob/main/src/LLM_pipeline/util/dataset.py"
    Data --> Preproc
    Preproc --> Loader
    Loader --> RP

    %% Classification Module
    subgraph "Classification Module"
        GPTC["gpt_classifier.py"]:::core
        LLmC["llama_classifier.py"]:::core
        CU["classifierutil.py"]:::core
        CPT["Prompts (classifier)"]:::cache
        MR["report_metrics.py"]:::core
    end
    click GPTC "https://github.com/arashdn/tabulax/blob/main/src/LLM_pipeline/classifier/gpt_classifier.py"
    click LLmC "https://github.com/arashdn/tabulax/blob/main/src/LLM_pipeline/classifier/llama_classifier.py"
    click CU "https://github.com/arashdn/tabulax/blob/main/src/LLM_pipeline/classifier/classifierutil.py"
    click CPT "https://github.com/arashdn/tabulax/tree/main/src/LLM_pipeline/classifier/prompts"
    click MR "https://github.com/arashdn/tabulax/blob/main/src/LLM_pipeline/classifier/report_metrics.py"

    RP -->|classification_type == gpt| GPTC
    RP -->|classification_type == llama| LLmC
    CPT --> GPTC
    CPT --> LLmC
    GPTC --> CU
    LLmC --> CU

    %% Transformation Module
    subgraph "Transformation Module"
        BasicT["basic.py"]:::core
        AlgoT["algorithmic.py"]:::core
        GenT["general.py"]:::core
        NumT["numeric.py"]:::core
        StrT["string.py"]:::core
        TPT["Prompts (transformer)"]:::cache
    end
    click BasicT "https://github.com/arashdn/tabulax/blob/main/src/LLM_pipeline/transformers/basic.py"
    click AlgoT "https://github.com/arashdn/tabulax/blob/main/src/LLM_pipeline/transformers/algorithmic.py"
    click GenT "https://github.com/arashdn/tabulax/blob/main/src/LLM_pipeline/transformers/general.py"
    click NumT "https://github.com/arashdn/tabulax/blob/main/src/LLM_pipeline/transformers/numeric.py"
    click StrT "https://github.com/arashdn/tabulax/blob/main/src/LLM_pipeline/transformers/string.py"
    click TPT "https://github.com/arashdn/tabulax/tree/main/src/LLM_pipeline/transformers/prompts"

    CU -->|class labels| BasicT
    CU --> AlgoT
    CU --> GenT
    CU --> NumT
    CU --> StrT
    TPT --> BasicT
    TPT --> AlgoT
    TPT --> GenT
    TPT --> NumT
    TPT --> StrT

    %% Cache Layer
    subgraph "Cache Layer"
        PC["Prompt Cache"]:::cache
        EC["ED Cache"]:::cache
    end
    CPT --> PC
    TPT --> PC
    BasicT --> PC
    AlgoT --> PC
    GenT --> PC
    NumT --> PC
    StrT --> PC

    %% LLM Service
    subgraph "LLM Service"
        OA["OpenAI API"]:::external
        VLLM["vLLM Server"]:::external
    end
    CU -->|prompt| OA
    CU -->|prompt| VLLM
    BasicT -->|prompt| OA
    BasicT -->|prompt| VLLM
    AlgoT -->|prompt| OA
    AlgoT -->|prompt| VLLM
    GenT -->|prompt| OA
    GenT -->|prompt| VLLM
    NumT -->|prompt| OA
    NumT -->|prompt| VLLM
    StrT -->|prompt| OA
    StrT -->|prompt| VLLM

    OA -->|response| GPTC
    VLLM -->|response| LLmC
    OA -->|response| BasicT
    VLLM -->|response| BasicT
    OA -->|response| AlgoT
    VLLM -->|response| AlgoT
    OA -->|response| GenT
    VLLM -->|response| GenT
    OA -->|response| NumT
    VLLM -->|response| NumT
    OA -->|response| StrT
    VLLM -->|response| StrT

    %% Output and Evaluation
    Output["OUTPUT_DIR (transformed CSVs, reports)"]:::data
    subgraph "Evaluation & Reporting"
        JE["JoinEval.py"]:::core
        MR
    end
    click JE "https://github.com/arashdn/tabulax/blob/main/src/LLM_pipeline/util/JoinEval.py"

    BasicT -->|transformed CSV| Output
    AlgoT --> Output
    GenT --> Output
    NumT --> Output
    StrT --> Output

    Output --> JE
    JE --> Output
    MR --> Output

    RP --> Output

    %% Styles
    classDef data fill:#CDF7CD,stroke:#333,stroke-width:1px
    classDef core fill:#D0E8FF,stroke:#333,stroke-width:1px
    classDef external fill:#FFD5B4,stroke:#333,stroke-width:1px
    classDef cache fill:#E0E0E0,stroke:#333,stroke-width:1px
    classDef config fill:#E8F0D0,stroke:#333,stroke-width:1px
```
