# sohomx/simtest architecture

> Architecture diagram of the GitHub repository sohomx/simtest, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sohomx/simtest
- Repository: https://github.com/sohomx/simtest

## Overview

##### Project Type & Purpose
- SimTest is an open-source CLI tool and test harness for fuzz-testing DAG-style LLM agents.
- It loads or imports an agent graph, generates seeds, runs them in a sandbox, and outputs cost/coverage/latency/policy reports.

##### Main Architectural Components
a. CLI Layer
- simtest/cli.py and __main__.py define subcommands: init, seed, import, fuzz, report.
b. Graph & Seed Management
- Core graph builder: simtest/core/loader.py + graph_builder.py
- Seed generator: simtest/seedgen/generator.py
- Seed loader: simtest/seeds/loader.py + YAML seeds in seeds/
c. Importers
- Trace import: simtest/importer/langsmith.py and trace.py
- Produces .simgraph.json and generates seed packs.
d. Fuzz Execution
- simtest/fuzz/sandbox.py wraps a sandboxed Python process.
- simtest/fuzz/tracker.py tracks per-node cost & latency.
e. Coverage & Verdict
- Coverage calculation: simtest/coverage/calc.py
- Verdict engine (schema, exceptions, policy): simtest/verdict/engine.py, judge.py
f. Reporting
- Markdown writer: simtest/report/writer.py
- Jinja templates for tables and custom sections (jinja_templates/)
g. Scripts & CI Integrations
- tools/seed_builder.py and scripts/qa_filter.py for seed pipelines.
- GitHub Actions workflows under .github/ for CI gates.

##### Component Interactions & Data Flows
- User invokes `simtest <command>` → CLI parses flags → routes to submodule.
- `init` → loader → graph_builder → writes .simgraph.json.
- `seed` / `generate` → seedgen → writes YAML or JSONL seeds.
- `import` → importer → loads trace → builder → .simgraph.json + seeds.
- `fuzz` → loads .simgraph.json + seeds → launches sandbox runner per seed → tracker collects metrics → verdict engine applies schema/policy/cost checks → coverage module aggregates coverage.
- `report` → report.writer uses collected metrics + Jinja templates → produces markdown.
- CI Action ties `pip install .` + `simtest fuzz ...` → fail/pass based on thresholds.

##### Architectural Patterns & Design Principles
- Modular CLI with subcommands (separation of concerns).
- Plugin-style importers and seed generators (extensible for new DAG frameworks).
- Clear separation: graph representation, seed management, execution sandbox, metrics, reporting.
- Zero-config by convention (auto-detect .simgraph.json, seeds/).
- Side-effect-free sandbox for safe LLM calls and reproducibility.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Inputs & Configuration"
        AgentCode["Agent Code (examples/)"]:::input
        TraceFile["Trace File (.jsonl)"]:::input
        RawSeeds["Raw Seeds (.jsonl)"]:::input
        YAMLSeeds["Seed Packs (seeds/)"]:::artifact
        Env["OPENAI_API_KEY"]:::external
    end

    CLI["simtest <command>"]:::cli
    click CLI "https://github.com/sohomx/simtest/blob/main/simtest/__main__.py"
    click CLI "https://github.com/sohomx/simtest/blob/main/simtest/cli.py"

    subgraph "Core Engine"
        subgraph "Importers"
            LangsmithImporter["Langsmith Importer"]:::core
            TraceImporter["Trace Importer"]:::core
        end
        Loader["Loader"]:::core
        GraphBuilder["Graph Builder"]:::core
        SimGraph[".simgraph.json"]:::artifact
        SeedGen["Seed Generator"]:::core
        SeedLoader["Seed Loader"]:::core
        subgraph "Fuzz Execution"
            FuzzRunner["Fuzz Runner (Sandbox)"]:::core
            Tracker["Tracker"]:::core
        end
        Coverage["Coverage Calculator"]:::core
        Verdict["Verdict Engine"]:::core
        Judge["Judge Logic"]:::core
    end

    subgraph "Scripts & CI Integrations"
        SeedBuilderTool["Seed Builder Tool"]:::script
        QAFilter["QA Filter Script"]:::script
        CI["CI Gate (GitHub Actions)"]:::external
    end

    subgraph "Outputs & Integrations"
        Report["Markdown Report"]:::artifact
        Jinja["Jinja Templates"]:::artifact
        PyPI["PyPI Distribution (dist/)"]:::artifact
    end

    %% Data Flows
    AgentCode -->|reads| CLI
    TraceFile -->|reads| CLI
    RawSeeds -->|reads| CLI
    YAMLSeeds -->|reads| CLI
    Env -->|provides creds| CLI

    CLI -->|"init/import"| LangsmithImporter
    CLI -->|"init/import"| TraceImporter
    LangsmithImporter -->|produces| SimGraph
    TraceImporter -->|produces| SimGraph
    LangsmithImporter -->|generates| YAMLSeeds
    TraceImporter -->|generates| YAMLSeeds

    CLI -->|"init"| Loader
    Loader -->|"builds graph"| GraphBuilder
    GraphBuilder -->|writes| SimGraph

    CLI -->|"seed"| SeedGen
    SeedGen -->|writes| YAMLSeeds

    CLI -->|"load seeds"| SeedLoader
    SeedLoader -->|reads| YAMLSeeds

    SimGraph -->|reads| FuzzRunner
    YAMLSeeds -->|reads| FuzzRunner
    FuzzRunner -->|executes| Tracker

    Tracker -->|metrics| Coverage
    Tracker -->|metrics| Verdict
    Verdict -->|verdicts| Judge

    Coverage -->|data for| Report
    Verdict -->|data for| Report
    Jinja -->|templates for| Report

    Report -->|publishes| PyPI
    Report -->|posts status| CI

    CLI -->|runs| SeedBuilderTool
    CLI -->|runs| QAFilter
    CI -->|invokes| CLI

    %% Click Events
    click LangsmithImporter "https://github.com/sohomx/simtest/blob/main/simtest/importer/langsmith.py"
    click TraceImporter "https://github.com/sohomx/simtest/blob/main/simtest/importer/trace.py"
    click Loader "https://github.com/sohomx/simtest/blob/main/simtest/core/loader.py"
    click GraphBuilder "https://github.com/sohomx/simtest/blob/main/simtest/core/graph_builder.py"
    click SeedGen "https://github.com/sohomx/simtest/blob/main/simtest/seedgen/generator.py"
    click SeedLoader "https://github.com/sohomx/simtest/blob/main/simtest/seeds/loader.py"
    click FuzzRunner "https://github.com/sohomx/simtest/blob/main/simtest/fuzz/sandbox.py"
    click Tracker "https://github.com/sohomx/simtest/blob/main/simtest/fuzz/tracker.py"
    click Coverage "https://github.com/sohomx/simtest/blob/main/simtest/coverage/calc.py"
    click Verdict "https://github.com/sohomx/simtest/blob/main/simtest/verdict/engine.py"
    click Judge "https://github.com/sohomx/simtest/blob/main/simtest/verdict/judge.py"
    click Jinja "https://github.com/sohomx/simtest/tree/main/jinja_templates/"
    click SeedBuilderTool "https://github.com/sohomx/simtest/blob/main/tools/seed_builder.py"
    click QAFilter "https://github.com/sohomx/simtest/blob/main/scripts/qa_filter.py"
    click CI "https://github.com/sohomx/simtest/blob/main/.github/workflows/simtest.yml"

    %% Styles
    classDef input fill:#D0E6F8,stroke:#0366D6,color:#0366D6;
    classDef core fill:#E6F4EA,stroke:#1B8A1B,color:#1B8A1B;
    classDef artifact fill:#FFF5BA,stroke:#E3A600,color:#E3A600;
    classDef external fill:#F0F0F0,stroke:#A0A0A0,color:#505050;
    classDef cli fill:#FDE2E4,stroke:#D93025,color:#D93025,stroke-width:2px;
    classDef script fill:#FFE0B2,stroke:#FF8C00,color:#FF8C00;
```
