How samholt/hdtwingen works
Main Components
• Configuration Manager
• Module: config/config.yaml + code in config/
• Responsibility: reads run parameters, datasets, baselines.
• Orchestrator / Runner
• Script: run.py
• Calls: executor.py to drive the evolutionary experiments.
• Evolutionary Engine (HDTwinGen core)
• Modules: agents.py, executor.py
• Orchestrates propose–evaluate–select loops.
• LLM Interface
• Modules: llm_utils.py, rate_limiter.py, utils/prompts.py
• Connects to external LLM API (e.g., OpenAI) to generate and refine hybrid model specs.
• Model Synthesizer & Modifier
• Module: summarizer_and_modifier.py
• Uses LLM output to assemble a hybrid digital twin specification.
• Offline Parameter Optimizer
• Modules in utils/: gp_method.py, sindy_utils.py, utils/models.py
• Calibrates mechanistic & neural components on data.
• Simulation & Evaluation
• Script: simulate.py, modules in libs/ (cancer/, covasim/, covid_cgp/)
• Runs candidate models against datasets to compute fitness metrics.
• Data Pipeline
• Modules: libs/covid_cgp/data_loader.py, libs/*/data_utils.py
• Loads time‐series and policy/demographic features from “libs/” subfolders.
• Result Processing & Reporting
• Script: process_result_file.py, utils/plot_utils.py, utils/results_utils.py
• Transforms logs into tables & figures.
• Utilities & Logging
• Modules: utils/logging_utils.py, utils/checkpoint_utils.py, utils/exp_utils.py
Relationships & Data Flows
• Start: Configuration Manager → Orchestrator (run.py)
• Orchestrator invokes Evolutionary Engine
– Each generation:
· Evolutionary Engine → LLM Interface (propose new specs)
· LLM Interface → Model Synthesizer → Offline Optimizer → Simulation & Evaluation
· Simulation returns fitness → Evolutionary Engine (selection)
• Loop until convergence or budget
• Final: Orchestrator → Result Processing → Report
• Data Sources (external CSVs, PDFs) feed into Data Pipeline modules
• External service: LLM API
Architectural Patterns & Principles
• Modular Pipeline: clear separation of concerns (config, generation, evaluation, processing)
• Loop / Feedback Pattern: evolution is an iterative generate–evaluate cycle
• Plugin‐style domain back‐ends: “libs/” contains interchangeable dataset/simulator modules
• Utility Layer: cross‐cutting concerns (logging, checkpointing, plotting)
Technologies & Frameworks
• Python 3
• YAML for configuration
• External LLM API (e.g., OpenAI) via llm_utils + rate limiter
• Evolutionary algorithm orchestrated in pure-Python (agents.py, executor.py)
• Domain libraries: covasim, cancer, covid_cgp for mechanistic simulation
• Data files in repos under libs/ (CSV, Excel, PDF)
Detail Level
• Essential modules listed above – do not over-split; group related scripts under “Offline Optimizer” or “Data Pipeline” when appropriate