# humanface-tech/nichebench architecture

> Architecture diagram of the GitHub repository humanface-tech/nichebench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/humanface-tech/nichebench
- Repository: https://github.com/humanface-tech/nichebench

## Overview

NicheBench is a CLI harness for benchmarking LLMs on framework-specific tasks, from static quiz and code tasks to agentic Drupal runtime tasks. The main workflow is to select or discover tasks, execute a model-under-test run, evaluate its output with deterministic checks and/or an LLM judge, then persist results for reporting and forensics. Runtime tasks add a live DDEV workspace, container execution, artifact checks, and trajectory capture. The map follows README capabilities and sampled modules; some unsampled subsystem wiring is intentionally left out.

Main entry points: [`src/nichebench/cli/app.py`](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/cli/app.py) (CLI commands), [`src/nichebench/execution/orchestrator.py`](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/orchestrator.py) (Test orchestrator), [`src/nichebench/execution/runtime/executor/flow.py`](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runtime/executor/flow.py) (Runtime stages), [`src/nichebench/execution/runners/mut.py`](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runners/mut.py) (Model task runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_cli["CLI and discovery"]
  node_cli_app["CLI commands<br/>[app.py]"]
  node_discovery["Framework discovery<br/>[discovery.py]"]
  node_task_loader["Task manifests"]
  node_config["Configuration"]
end

subgraph group_eval["Benchmark evaluation"]
  node_orchestrator["Test orchestrator<br/>[orchestrator.py]"]
  node_mut_runner["Model task runner<br/>[mut.py]"]
  node_agent["Code agent<br/>[agent.py]"]
  node_llm_client["LLM client<br/>[litellm_client.py]"]
  node_judge_runner["Judge runner<br/>[judge.py]"]
  node_judge_adapter["Judge adapter<br/>[judge.py]"]
  node_metrics["Task metrics<br/>[quiz_metric.py]"]
end

subgraph group_runtime["Runtime execution"]
  node_runtime_executor["Runtime stages<br/>[flow.py]"]
  node_cage["Container execution<br/>[mixin.py]"]
  node_workspace["DDEV workspace<br/>[ddev.py]"]
  node_scorer["Runtime scoring<br/>[scorer.py]"]
  node_trajectory["Trajectory capture<br/>[session_files.py]"]
  node_trajectory_db[("Trajectory store<br/>[sqlite.py]")]
end

subgraph group_results["Results and diagnostics"]
  node_persistence["Run persistence<br/>[persistence.py]"]
  node_summary["Run summaries<br/>[summary.py]"]
  node_forensics["Forensics reports<br/>[forensics.py]"]
  node_report_cli["Report views<br/>[report.py]"]
end

node_user(("Benchmark user"))
node_model_service["LLM service"]
node_ddev_service["Docker and DDEV"]

node_user -->|"invokes"| node_cli_app
node_cli_app -->|"lists tasks"| node_discovery
node_cli_app -->|"shows cases"| node_task_loader
node_cli_app -->|"loads settings"| node_config
node_cli_app -->|"starts runs"| node_orchestrator
node_orchestrator -->|"loads tasks"| node_task_loader
node_orchestrator -->|"runs model"| node_mut_runner
node_mut_runner -.->|"dispatches agent"| node_agent
node_mut_runner -->|"requests generation"| node_llm_client
node_llm_client -->|"calls model API"| node_model_service
node_orchestrator -->|"requests judging"| node_judge_runner
node_judge_runner -->|"evaluates output"| node_metrics
node_judge_runner -->|"requests verdict"| node_judge_adapter
node_judge_adapter -->|"generates verdict"| node_llm_client
node_runtime_executor -->|"runs container task"| node_cage
node_cage -->|"uses workspace"| node_workspace
node_cage -->|"starts containers"| node_ddev_service
node_runtime_executor -->|"scores artifacts"| node_scorer
node_runtime_executor -->|"collects session"| node_trajectory
node_trajectory -->|"stores trajectory"| node_trajectory_db
node_orchestrator -->|"dispatches runtime"| node_runtime_executor
node_orchestrator -->|"saves results"| node_persistence
node_persistence -->|"updates summary"| node_summary
node_user -->|"requests reports"| node_report_cli
node_report_cli -->|"reads run data"| node_persistence
node_cli_app -->|"requests analysis"| node_forensics

click node_cli_app "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/cli/app.py"
click node_discovery "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/core/discovery.py"
click node_task_loader "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/core/framework_loader.py"
click node_config "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/config/nichebench_config.py"
click node_orchestrator "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/orchestrator.py"
click node_mut_runner "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runners/mut.py"
click node_agent "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/providers/langgraph_code_agent/agent.py"
click node_llm_client "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/providers/litellm_client.py"
click node_judge_runner "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runners/judge.py"
click node_judge_adapter "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/providers/litellm_judge/judge.py"
click node_metrics "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/metrics/quiz_metric.py"
click node_runtime_executor "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runtime/executor/flow.py"
click node_cage "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runtime/cage/mixin.py"
click node_workspace "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runtime/workspace/ddev.py"
click node_scorer "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runtime/scoring/scorer.py"
click node_trajectory "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runtime/trajectory/session_files.py"
click node_trajectory_db "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runtime/trajectory/sqlite.py"
click node_persistence "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/persistence.py"
click node_summary "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/summary.py"
click node_forensics "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/diagnostics/forensics.py"
click node_report_cli "https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/cli/commands/report.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli_app,node_discovery,node_task_loader,node_config,node_user toneBlue
class node_orchestrator,node_mut_runner,node_agent,node_llm_client,node_judge_runner,node_judge_adapter,node_metrics toneAmber
class node_runtime_executor,node_cage,node_workspace,node_scorer,node_trajectory,node_trajectory_db toneMint
class node_persistence,node_summary,node_forensics,node_report_cli toneRose
class node_model_service,node_ddev_service toneIndigo
```

## Components

### CLI and discovery

- [CLI commands](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/cli/app.py): `src/nichebench/cli/app.py`
- [Framework discovery](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/core/discovery.py): `src/nichebench/core/discovery.py`
- [Task manifests](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/core/framework_loader.py): `src/nichebench/core/framework_loader.py`
- [Configuration](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/config/nichebench_config.py): `src/nichebench/config/nichebench_config.py`

### Benchmark evaluation

- [Test orchestrator](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/orchestrator.py): `src/nichebench/execution/orchestrator.py`
- [Model task runner](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runners/mut.py): `src/nichebench/execution/runners/mut.py`
- [Code agent](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/providers/langgraph_code_agent/agent.py): `src/nichebench/providers/langgraph_code_agent/agent.py`
- [LLM client](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/providers/litellm_client.py): `src/nichebench/providers/litellm_client.py`
- [Judge runner](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runners/judge.py): `src/nichebench/execution/runners/judge.py`
- [Judge adapter](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/providers/litellm_judge/judge.py): `src/nichebench/providers/litellm_judge/judge.py`
- [Task metrics](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/metrics/quiz_metric.py): `src/nichebench/metrics/quiz_metric.py`

### Runtime execution

- [Runtime stages](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runtime/executor/flow.py): `src/nichebench/execution/runtime/executor/flow.py`
- [Container execution](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runtime/cage/mixin.py): `src/nichebench/execution/runtime/cage/mixin.py`
- [DDEV workspace](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runtime/workspace/ddev.py): `src/nichebench/execution/runtime/workspace/ddev.py`
- [Runtime scoring](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runtime/scoring/scorer.py): `src/nichebench/execution/runtime/scoring/scorer.py`
- [Trajectory capture](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runtime/trajectory/session_files.py): `src/nichebench/execution/runtime/trajectory/session_files.py`
- [Trajectory store](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/runtime/trajectory/sqlite.py): `src/nichebench/execution/runtime/trajectory/sqlite.py`

### Results and diagnostics

- [Run persistence](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/persistence.py): `src/nichebench/execution/persistence.py`
- [Run summaries](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/summary.py): `src/nichebench/execution/summary.py`
- [Forensics reports](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/execution/diagnostics/forensics.py): `src/nichebench/execution/diagnostics/forensics.py`
- [Report views](https://github.com/humanface-tech/nichebench/blob/main/src/nichebench/cli/commands/report.py): `src/nichebench/cli/commands/report.py`

### Other components

- **Benchmark user**
- **LLM service**
- **Docker and DDEV**

## Connections

- Benchmark user → CLI commands: invokes
- CLI commands → Framework discovery: lists tasks
- CLI commands → Task manifests: shows cases
- CLI commands → Configuration: loads settings
- CLI commands → Test orchestrator: starts runs
- Test orchestrator → Task manifests: loads tasks
- Test orchestrator → Model task runner: runs model
- Model task runner → Code agent: dispatches agent
- Model task runner → LLM client: requests generation
- LLM client → LLM service: calls model API
- Test orchestrator → Judge runner: requests judging
- Judge runner → Task metrics: evaluates output
- Judge runner → Judge adapter: requests verdict
- Judge adapter → LLM client: generates verdict
- Runtime stages → Container execution: runs container task
- Container execution → DDEV workspace: uses workspace
- Container execution → Docker and DDEV: starts containers
- Runtime stages → Runtime scoring: scores artifacts
- Runtime stages → Trajectory capture: collects session
- Trajectory capture → Trajectory store: stores trajectory
- Test orchestrator → Runtime stages: dispatches runtime
- Test orchestrator → Run persistence: saves results
- Run persistence → Run summaries: updates summary
- Benchmark user → Report views: requests reports
- Report views → Run persistence: reads run data
- CLI commands → Forensics reports: requests analysis