How humanface-tech/nichebench works
NicheBench is a CLI harness for benchmarking LLMs on framework-specific tasks, from static quiz and code tasks to agentic Drupal runtime tasks. The main workflow is to select or discover tasks, execute a model-under-test run, evaluate its output with deterministic checks and/or an LLM judge, then persist results for reporting and forensics. Runtime tasks add a live DDEV workspace, container execution, artifact checks, and trajectory capture. The map follows README capabilities and sampled modules; some unsampled subsystem wiring is intentionally left out.
Main entry points: app.py CLI commands · orchestrator.py Test orchestrator · flow.py Runtime stages · mut.py Model task runner
Components 24
CLI and discovery
- CLI commands
src/nichebench/cli/app.py - Framework discovery
src/nichebench/core/discovery.py - Task manifests
src/nichebench/core/framework_loader.py - Configuration
src/nichebench/config/nichebench_config.py
Benchmark evaluation
- Test orchestrator
src/nichebench/execution/orchestrator.py - Model task runner
src/nichebench/execution/runners/mut.py - Code agent
src/nichebench/providers/langgraph_code_agent/agent.py - LLM client
src/nichebench/providers/litellm_client.py - Judge runner
src/nichebench/execution/runners/judge.py - Judge adapter
src/nichebench/providers/litellm_judge/judge.py - Task metrics
src/nichebench/metrics/quiz_metric.py
Runtime execution
- Runtime stages
src/nichebench/execution/runtime/executor/flow.py - Container execution
src/nichebench/execution/runtime/cage/mixin.py - DDEV workspace
src/nichebench/execution/runtime/workspace/ddev.py - Runtime scoring
src/nichebench/execution/runtime/scoring/scorer.py - Trajectory capture
src/nichebench/execution/runtime/trajectory/session_files.py - Trajectory store
src/nichebench/execution/runtime/trajectory/sqlite.py
Results and diagnostics
- Run persistence
src/nichebench/execution/persistence.py - Run summaries
src/nichebench/execution/summary.py - Forensics reports
src/nichebench/execution/diagnostics/forensics.py - Report views
src/nichebench/cli/commands/report.py
Other components
- Benchmark user
- LLM service
- Docker and DDEV
Connections 26
- Benchmark user → CLI commands: invokes
- CLI commands → Framework discovery: lists tasks
- CLI commands → Task manifests: shows cases
- CLI commands → Configuration: loads settings
- CLI commands → Test orchestrator: starts runs
- Test orchestrator → Task manifests: loads tasks
- Test orchestrator → Model task runner: runs model
- Model task runner → Code agent: dispatches agent
- Model task runner → LLM client: requests generation
- LLM client → LLM service: calls model API
- Test orchestrator → Judge runner: requests judging
- Judge runner → Task metrics: evaluates output
- Judge runner → Judge adapter: requests verdict
- Judge adapter → LLM client: generates verdict
- Runtime stages → Container execution: runs container task
- Container execution → DDEV workspace: uses workspace
- Container execution → Docker and DDEV: starts containers
- Runtime stages → Runtime scoring: scores artifacts
- Runtime stages → Trajectory capture: collects session
- Trajectory capture → Trajectory store: stores trajectory
- Test orchestrator → Runtime stages: dispatches runtime
- Test orchestrator → Run persistence: saves results
- Run persistence → Run summaries: updates summary
- Benchmark user → Report views: requests reports
- Report views → Run persistence: reads run data
- CLI commands → Forensics reports: requests analysis