# samholt/hdtwingen architecture

> Architecture diagram of the GitHub repository samholt/hdtwingen, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/samholt/hdtwingen
- Repository: https://github.com/samholt/hdtwingen

## Overview

##### Main Components
- Configuration Manager
- Module: config/config.yaml + code in config/
- Responsibility: reads run parameters, datasets, baselines.
- Orchestrator / Runner
- Script: run.py
- Calls: executor.py to drive the evolutionary experiments.
- Evolutionary Engine (HDTwinGen core)
- Modules: agents.py, executor.py
- Orchestrates propose–evaluate–select loops.
- LLM Interface
- Modules: llm_utils.py, rate_limiter.py, utils/prompts.py
- Connects to external LLM API (e.g., OpenAI) to generate and refine hybrid model specs.
- Model Synthesizer & Modifier
- Module: summarizer_and_modifier.py
- Uses LLM output to assemble a hybrid digital twin specification.
- Offline Parameter Optimizer
- Modules in utils/: gp_method.py, sindy_utils.py, utils/models.py
- Calibrates mechanistic & neural components on data.
- Simulation & Evaluation
- Script: simulate.py, modules in libs/ (cancer/, covasim/, covid_cgp/)
- Runs candidate models against datasets to compute fitness metrics.
- Data Pipeline
- Modules: libs/covid_cgp/data_loader.py, libs/*/data_utils.py
- Loads time‐series and policy/demographic features from “libs/” subfolders.
- Result Processing & Reporting
- Script: process_result_file.py, utils/plot_utils.py, utils/results_utils.py
- Transforms logs into tables & figures.
- Utilities & Logging
- Modules: utils/logging_utils.py, utils/checkpoint_utils.py, utils/exp_utils.py

##### Relationships & Data Flows
- Start: Configuration Manager → Orchestrator (run.py)
- Orchestrator invokes Evolutionary Engine
– Each generation:
· Evolutionary Engine → LLM Interface (propose new specs)
· LLM Interface → Model Synthesizer → Offline Optimizer → Simulation & Evaluation
· Simulation returns fitness → Evolutionary Engine (selection)
- Loop until convergence or budget
- Final: Orchestrator → Result Processing → Report
- Data Sources (external CSVs, PDFs) feed into Data Pipeline modules
- External service: LLM API

##### Architectural Patterns & Principles
- Modular Pipeline: clear separation of concerns (config, generation, evaluation, processing)
- Loop / Feedback Pattern: evolution is an iterative generate–evaluate cycle
- Plugin‐style domain back‐ends: “libs/” contains interchangeable dataset/simulator modules
- Utility Layer: cross‐cutting concerns (logging, checkpointing, plotting)

##### Technologies & Frameworks
- Python 3
- YAML for configuration
- External LLM API (e.g., OpenAI) via llm_utils + rate limiter
- Evolutionary algorithm orchestrated in pure-Python (agents.py, executor.py)
- Domain libraries: covasim, cancer, covid_cgp for mechanistic simulation
- Data files in repos under libs/ (CSV, Excel, PDF)

##### Detail Level
- Essential modules listed above – do not over-split; group related scripts under “Offline Optimizer” or “Data Pipeline” when appropriate

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Main Flow
    Config["Configuration Manager"]:::config
    Runner["Orchestrator / Runner"]:::core
    Engine["Evolutionary Engine"]:::core
    LLMInterface["LLM Interface"]:::external
    Synthesizer["Model Synthesizer & Modifier"]:::core
    OfflineOpt["Offline Parameter Optimizer"]:::core
    Simulation["Simulation & Evaluation"]:::domain
    ResultProc["Result Processing & Reporting"]:::report
    DataPipeline["Data Pipeline"]:::domain
    DataSources[(External Data Files)]:::data
    LLMAPI[(LLM API)]:::external

    %% Flow Connections
    Config -->|reads params| Runner
    Runner -->|starts loop| Engine

    subgraph "Evolutionary Loop"
        direction TB
        Engine -->|propose specs| LLMInterface
        LLMInterface -->|raw specs| Synthesizer
        Synthesizer -->|tuned specs| OfflineOpt
        OfflineOpt -->|model spec → simulation| Simulation
        Simulation -->|fitness metrics| Engine
    end

    Runner -->|final results| ResultProc

    DataSources -->|time-series, features| DataPipeline
    DataPipeline -->|data feed| Simulation
    LLMInterface -->|API calls| LLMAPI

    %% Click Events
    click Config "https://github.com/samholt/hdtwingen/blob/master/config/config.yaml"
    click Runner "https://github.com/samholt/hdtwingen/blob/master/run.py"
    click Engine "https://github.com/samholt/hdtwingen/blob/master/executor.py"
    click LLMInterface "https://github.com/samholt/hdtwingen/blob/master/llm_utils.py"
    click Synthesizer "https://github.com/samholt/hdtwingen/blob/master/summarizer_and_modifier.py"
    click OfflineOpt "https://github.com/samholt/hdtwingen/blob/master/utils/gp_method.py"
    click Simulation "https://github.com/samholt/hdtwingen/blob/master/simulate.py"
    click DataPipeline "https://github.com/samholt/hdtwingen/blob/master/libs/covid_cgp/data_loader.py"
    click ResultProc "https://github.com/samholt/hdtwingen/blob/master/process_result_file.py"
    click UtilsLog "https://github.com/samholt/hdtwingen/blob/master/utils/logging_utils.py"

    %% Utility & Logging (cross-cutting)
    UtilsLog["Utilities & Logging"]:::util
    UtilsLog -.-> Runner
    UtilsLog -.-> Engine
    UtilsLog -.-> ResultProc

    %% Styles
    classDef core fill:#cfe2f3,stroke:#000,stroke-width:1px;
    classDef domain fill:#d9ead3,stroke:#000,stroke-width:1px;
    classDef external fill:#f9cb9c,stroke:#000,stroke-width:1px;
    classDef config fill:#fff2cc,stroke:#000,stroke-width:1px;
    classDef report fill:#ead1dc,stroke:#000,stroke-width:1px;
    classDef util fill:#d0d0d0,stroke:#000,stroke-width:1px;
```
