How iaar-shanghai/datg works
Project type and purpose
• This is a research‐oriented framework for controlled text generation (DATG) using large language models (LLMs).
• It’s not a web or full‐stack app; it’s a command-line/CLI‐driven experiment pipeline with modular “methods” plugins (DATG, FUDGE, PREADD), plus training, evaluation, and stats generation.
File structure
• Top-level “main.py” and “analyst.py” drive the core workflows: experiments and statistics.
• “config.py” holds experiment settings: model paths, task definitions, classifier locations, API keys.
• “method/” contains pluggable CTG modules: datg/, fudge/, preadd/, each with their own logic and internal classifier helpers.
• “data/” holds raw datasets (ori_data/…), processed classifier training data (internal_classifier_data/…), external evaluation data (external_classifier_data/…).
• “train/” notebooks and “train_config.py” produce and save classifier models into data/internal_classifier_data.
• “utils/” provides reusable code: classifier wrappers, metrics, and NLP utilities (nltk corpora).
• “results/” captures JSON outputs per model/task run. “stats/” stores aggregated CSV stats.
• “docs/” and “assets/” host static documentation and images for project overview.
README insights
• The pipeline: configure → run experiments → generate statistics.
• CTG methods rely on dynamic attribute graphs to guide decoding.
• External classifiers (for evaluation) vs. internal classifiers (for guiding generation).
• LLMs are loaded via MODEL_PATHS (HuggingFace or converted models).
• Optional external Perspective API used for toxicity scoring.
Components
a. Input/CLI layer
– “main.py” CLI: accepts --model_name and --task_name
– “analyst.py” CLI: post‐hoc stats generation
b. Configuration
– “config.py”: experiment parameters, model paths, API keys
c. Data layer (data stores)
– Raw datasets: data/ori_data/…
– Classifier training data: data/internal_classifier_data/…
– Evaluation data: data/external_classifier_data/…
d. Training pipeline
– train/*.ipynb & train_config.py → produce classifier models → save to data/internal_classifier_data
e. Experiment orchestration
– main.py orchestrates:
• Load config
• Load chosen LLM via HuggingFace
• Load the chosen CTG method module (method/datg, method/fudge, method/preadd)
• For internal guidance: use internal classifier wrappers in method/<method>/classifier.py and utils/classifier.py
• Generate dynamic attribute graphs (method/datg/datg.py) or alternative logic
• Perform controlled decoding with the LLM
• Optionally call Perspective API (external service) for toxicity feedback
f. Evaluation & results
– main.py writes generated outputs and evaluation metrics to results/*.json
– utils/metrics.py computes fluency, attribute scores
g. Statistics generation
– analyst.py reads results/*.json → aggregates via utils/metrics.py → outputs stats/*.csv
h. Documentation site (docs/)
Relationships and data flow
• Orchestrator → Data Loader → Raw/Evaluation data
• Orchestrator → Model Loader → LLM
• Orchestrator → Method Module → Attribute Graph Generator + Internal Classifier
• Method Module → LLM (decoding) → Generated text
• Generated text → External Classifier or Perspective API → Evaluation metrics
• Outputs → Results storage
• Results → Analyst → Stats storage
Architectural patterns and principles
• Pipeline/ETL-style flow (load → process → save → analyze)
• Plugin architecture for CTG methods (drop in new method under method/)
• Separation of concerns: config vs. orchestration vs. data vs. evaluation vs. stats
• Modular utilities for classification and metrics