How sohomx/simtest works
Project Type & Purpose
• SimTest is an open-source CLI tool and test harness for fuzz-testing DAG-style LLM agents.
• It loads or imports an agent graph, generates seeds, runs them in a sandbox, and outputs cost/coverage/latency/policy reports.
Main Architectural Components
a. CLI Layer
• simtest/cli.py and __main__.py define subcommands: init, seed, import, fuzz, report.
b. Graph & Seed Management
• Core graph builder: simtest/core/loader.py + graph_builder.py
• Seed generator: simtest/seedgen/generator.py
• Seed loader: simtest/seeds/loader.py + YAML seeds in seeds/
c. Importers
• Trace import: simtest/importer/langsmith.py and trace.py
• Produces .simgraph.json and generates seed packs.
d. Fuzz Execution
• simtest/fuzz/sandbox.py wraps a sandboxed Python process.
• simtest/fuzz/tracker.py tracks per-node cost & latency.
e. Coverage & Verdict
• Coverage calculation: simtest/coverage/calc.py
• Verdict engine (schema, exceptions, policy): simtest/verdict/engine.py, judge.py
f. Reporting
• Markdown writer: simtest/report/writer.py
• Jinja templates for tables and custom sections (jinja_templates/)
g. Scripts & CI Integrations
• tools/seed_builder.py and scripts/qa_filter.py for seed pipelines.
• GitHub Actions workflows under .github/ for CI gates.
Component Interactions & Data Flows
• User invokes simtest <command> → CLI parses flags → routes to submodule.
• init → loader → graph_builder → writes .simgraph.json.
• seed / generate → seedgen → writes YAML or JSONL seeds.
• import → importer → loads trace → builder → .simgraph.json + seeds.
• fuzz → loads .simgraph.json + seeds → launches sandbox runner per seed → tracker collects metrics → verdict engine applies schema/policy/cost checks → coverage module aggregates coverage.
• report → report.writer uses collected metrics + Jinja templates → produces markdown.
• CI Action ties pip install . + simtest fuzz ... → fail/pass based on thresholds.
Architectural Patterns & Design Principles
• Modular CLI with subcommands (separation of concerns).
• Plugin-style importers and seed generators (extensible for new DAG frameworks).
• Clear separation: graph representation, seed management, execution sandbox, metrics, reporting.
• Zero-config by convention (auto-detect .simgraph.json, seeds/).
• Side-effect-free sandbox for safe LLM calls and reproducibility.