# iaar-shanghai/datg architecture

> Architecture diagram of the GitHub repository iaar-shanghai/datg, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/iaar-shanghai/datg
- Repository: https://github.com/iaar-shanghai/datg

## Overview

##### Project type and purpose
- This is a research‐oriented framework for controlled text generation (DATG) using large language models (LLMs).
- It’s not a web or full‐stack app; it’s a command-line/CLI‐driven experiment pipeline with modular “methods” plugins (DATG, FUDGE, PREADD), plus training, evaluation, and stats generation.

##### File structure
- Top-level “main.py” and “analyst.py” drive the core workflows: experiments and statistics.
- “config.py” holds experiment settings: model paths, task definitions, classifier locations, API keys.
- “method/” contains pluggable CTG modules: datg/, fudge/, preadd/, each with their own logic and internal classifier helpers.
- “data/” holds raw datasets (ori_data/…), processed classifier training data (internal_classifier_data/…), external evaluation data (external_classifier_data/…).
- “train/” notebooks and “train_config.py” produce and save classifier models into data/internal_classifier_data.
- “utils/” provides reusable code: classifier wrappers, metrics, and NLP utilities (nltk corpora).
- “results/” captures JSON outputs per model/task run. “stats/” stores aggregated CSV stats.
- “docs/” and “assets/” host static documentation and images for project overview.

##### README insights
- The pipeline: configure → run experiments → generate statistics.
- CTG methods rely on dynamic attribute graphs to guide decoding.
- External classifiers (for evaluation) vs. internal classifiers (for guiding generation).
- LLMs are loaded via MODEL_PATHS (HuggingFace or converted models).
- Optional external Perspective API used for toxicity scoring.

##### Components
a. Input/CLI layer
– “main.py” CLI: accepts --model_name and --task_name
– “analyst.py” CLI: post‐hoc stats generation
b. Configuration
– “config.py”: experiment parameters, model paths, API keys
c. Data layer (data stores)
– Raw datasets: data/ori_data/…
– Classifier training data: data/internal_classifier_data/…
– Evaluation data: data/external_classifier_data/…
d. Training pipeline
– train/*.ipynb & train_config.py → produce classifier models → save to data/internal_classifier_data
e. Experiment orchestration
– main.py orchestrates:
- Load config
- Load chosen LLM via HuggingFace
- Load the chosen CTG method module (method/datg, method/fudge, method/preadd)
- For internal guidance: use internal classifier wrappers in method/<method>/classifier.py and utils/classifier.py
- Generate dynamic attribute graphs (method/datg/datg.py) or alternative logic
- Perform controlled decoding with the LLM
- Optionally call Perspective API (external service) for toxicity feedback
f. Evaluation & results
– main.py writes generated outputs and evaluation metrics to results/*.json
– utils/metrics.py computes fluency, attribute scores
g. Statistics generation
– analyst.py reads results/*.json → aggregates via utils/metrics.py → outputs stats/*.csv
h. Documentation site (docs/)

##### Relationships and data flow
- Orchestrator → Data Loader → Raw/Evaluation data
- Orchestrator → Model Loader → LLM
- Orchestrator → Method Module → Attribute Graph Generator + Internal Classifier
- Method Module → LLM (decoding) → Generated text
- Generated text → External Classifier or Perspective API → Evaluation metrics
- Outputs → Results storage
- Results → Analyst → Stats storage

##### Architectural patterns and principles
- Pipeline/ETL-style flow (load → process → save → analyze)
- Plugin architecture for CTG methods (drop in new method under method/)
- Separation of concerns: config vs. orchestration vs. data vs. evaluation vs. stats
- Modular utilities for classification and metrics

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Entrypoints
    main["main.py"]:::process
    analyst["analyst.py"]:::process

    %% Configuration
    config["config.py"]:::process

    %% Data Stores
    subgraph "Data Stores"
        raw["ori_data"]:::data
        internalData["internal_classifier_data"]:::data
        externalData["external_classifier_data"]:::data
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        nb1["sentiment_classifier.ipynb"]:::process
        nb2["sentiment_discriminator.ipynb"]:::process
        nb3["toxic_classifier.ipynb"]:::process
        trainConfig["train_config.py"]:::process
    end

    %% CTG Methods
    subgraph "CTG Methods"
        subgraph DATG
            datgLogic["datg.py"]:::process
            datgCls["classifier.py"]:::process
        end
        subgraph FUDGE
            fudgeLogic["fudge.py"]:::process
            fudgeCls["classifier.py"]:::process
            fudgeLog["logits_processor.py"]:::process
        end
        subgraph PREADD
            preaddLogic["preadd.py"]:::process
            preaddCls["classifier.py"]:::process
        end
    end

    %% Utilities
    subgraph "Utilities"
        clsWrapper["classifier.py"]:::process
        metrics["metrics.py"]:::process
        nltkData["nltk_data"]:::data
    end

    %% Results & Stats
    subgraph "Results & Stats"
        results["results"]:::data
        stats["stats"]:::data
    end

    %% Documentation
    subgraph "Documentation"
        docs["docs"]:::data
        assets["assets/"]:::data
    end

    %% External Services
    subgraph "External Services"
        llm["LLM Model Hub"]:::service
        perspective["Perspective API"]:::service
    end

    %% Connections
    main --> config
    analyst --> config

    main --> raw
    main --> externalData
    main --> llm
    main --> datgLogic
    main --> fudgeLogic
    main --> preaddLogic
    main --> perspective
    main --> metrics

    nb1 --> internalData
    nb2 --> internalData
    nb3 --> internalData
    trainConfig --> internalData

    datgLogic --> clsWrapper
    datgLogic --> internalData
    fudgeLogic --> clsWrapper
    fudgeLogic --> internalData
    preaddLogic --> clsWrapper
    preaddLogic --> internalData

    clsWrapper --> datgLogic
    clsWrapper --> fudgeLogic
    clsWrapper --> preaddLogic

    metrics --> main
    metrics --> analyst

    main --> results
    results --> analyst
    analyst --> stats

    %% Click Events
    click main "https://github.com/iaar-shanghai/datg/blob/main/main.py"
    click analyst "https://github.com/iaar-shanghai/datg/blob/main/analyst.py"
    click config "https://github.com/iaar-shanghai/datg/blob/main/config.py"
    click raw "https://github.com/iaar-shanghai/datg/tree/main/data/ori_data/"
    click internalData "https://github.com/iaar-shanghai/datg/tree/main/data/internal_classifier_data/"
    click externalData "https://github.com/iaar-shanghai/datg/tree/main/data/external_classifier_data/"
    click nb1 "https://github.com/iaar-shanghai/datg/blob/main/train/sentiment_classifier.ipynb"
    click nb2 "https://github.com/iaar-shanghai/datg/blob/main/train/sentiment_discriminator.ipynb"
    click nb3 "https://github.com/iaar-shanghai/datg/blob/main/train/toxic_classifier.ipynb"
    click trainConfig "https://github.com/iaar-shanghai/datg/blob/main/train/train_config.py"
    click datgLogic "https://github.com/iaar-shanghai/datg/blob/main/method/datg/datg.py"
    click datgCls "https://github.com/iaar-shanghai/datg/blob/main/method/datg/classifier.py"
    click fudgeLogic "https://github.com/iaar-shanghai/datg/blob/main/method/fudge/fudge.py"
    click fudgeCls "https://github.com/iaar-shanghai/datg/blob/main/method/fudge/classifier.py"
    click fudgeLog "https://github.com/iaar-shanghai/datg/blob/main/method/fudge/logits_processor.py"
    click preaddLogic "https://github.com/iaar-shanghai/datg/blob/main/method/preadd/preadd.py"
    click preaddCls "https://github.com/iaar-shanghai/datg/blob/main/method/preadd/classifier.py"
    click clsWrapper "https://github.com/iaar-shanghai/datg/blob/main/utils/classifier.py"
    click metrics "https://github.com/iaar-shanghai/datg/blob/main/utils/metrics.py"
    click results "https://github.com/iaar-shanghai/datg/tree/main/results/"
    click stats "https://github.com/iaar-shanghai/datg/tree/main/stats/"
    click docs "https://github.com/iaar-shanghai/datg/tree/main/docs/"
    click assets "https://github.com/iaar-shanghai/datg/tree/main/assets/"

    %% Styles
    classDef process fill:#f9f,stroke:#333,stroke-width:1px
    classDef data fill:#bbf,stroke:#333,stroke-width:1px
    classDef service fill:#bfb,stroke:#333,stroke-width:1px
```
