# yomileonel/beyondpass architecture

> Architecture diagram of the GitHub repository yomileonel/beyondpass, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yomileonel/beyondpass
- Repository: https://github.com/yomileonel/beyondpass

## Overview

BeyondPass evaluates LLM-generated solutions on HumanEval or MBPP tasks through a bounded Planner–Coder–Tester–Critic loop. The Critic compares candidates with references using AST-based structural scores, diagnoses failures, and supplies feedback for retries; the run also records results for comparison and visualization. The graph includes the standalone metrics package and optional dashboard. Some links rely on README or SOURCE INDEX references rather than inspected implementation excerpts, so unsampled internals are not expanded.

Main entry points: [`src/beyondpass/orchestrator.py`](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/orchestrator.py) (Iteration orchestrator), [`src/beyondpass/__main__.py`](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/__main__.py) (Run and report CLI), [`src/beyondpass/reporting/aggregate.py`](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/reporting/aggregate.py) (Run aggregation), [`src/beyondpass/agents/critic.py`](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/agents/critic.py) (Critic agent)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Run workflow"]
  node_cli["Run and report CLI<br/>[__main__.py]"]
  node_config["Run configuration<br/>[config.py]"]
  node_orchestrator["Iteration orchestrator<br/>[orchestrator.py]"]
  node_planner["Planner agent<br/>[planner.py]"]
  node_coder["Coder agent<br/>[coder.py]"]
  node_llm["LLM client<br/>[llm_client.py]"]
end

subgraph group_feedback["Structural feedback"]
  node_critic["Critic agent<br/>[critic.py]"]
  node_tokenizer["AST tokenizer<br/>[tokenizer.py]"]
  node_scores["Structural scores<br/>[scores.py]"]
  node_diagnosis["Failure diagnosis<br/>[diagnosis.py]"]
  node_templates["Feedback templates<br/>[templates.py]"]
  node_trivial["Trivial-solution flags<br/>[trivial.py]"]
end

subgraph group_evaluation["Benchmarks and isolation"]
  node_tester["Tester agent<br/>[tester.py]"]
  node_humaneval["HumanEval loader<br/>[humaneval.py]"]
  node_mbpp["MBPP loader<br/>[mbpp.py]"]
  node_sandbox["Docker sandbox<br/>[docker_runner.py]"]
end

subgraph group_results["Reporting"]
  node_aggregate["Run aggregation<br/>[aggregate.py]"]
  node_plots["Comparison plots<br/>[plots.py]"]
  node_dashboard["Interactive dashboard<br/>[dashboard.py]"]
  node_run_data["JSONL run results<br/>[orchestrator.py]"]
end

subgraph group_metrics_pkg["Standalone metrics"]
  node_pkg_tokenizer["Packaged tokenizer<br/>[tokenizer.py]"]
  node_pkg_scores["Packaged scores<br/>[scores.py]"]
end

node_user(("Engineer"))
node_openai["OpenAI API"]

node_user -->|"starts run"| node_cli
node_cli -->|"loads settings"| node_config
node_cli -->|"starts evaluation"| node_orchestrator
node_cli -->|"loads tasks"| node_humaneval
node_cli -->|"loads tasks"| node_mbpp
node_cli -->|"requests summary"| node_aggregate
node_cli -->|"renders comparison"| node_plots
node_orchestrator -->|"requests plan"| node_planner
node_orchestrator -->|"requests candidate"| node_coder
node_orchestrator -->|"tests candidate"| node_tester
node_orchestrator -->|"scores candidate"| node_critic
node_orchestrator -->|"builds feedback"| node_templates
node_orchestrator -->|"uses settings"| node_config
node_orchestrator -->|"writes results"| node_run_data
node_planner -->|"requests completion"| node_llm
node_coder -->|"requests completion"| node_llm
node_llm -->|"calls API"| node_openai
node_tester -->|"executes candidate"| node_sandbox
node_critic -->|"computes metrics"| node_scores
node_critic -->|"classifies failure"| node_diagnosis
node_critic -->|"checks input use"| node_trivial
node_scores -->|"tokenizes programs"| node_tokenizer
node_diagnosis -->|"uses thresholds"| node_config
node_aggregate -.->|"reads results"| node_run_data
node_aggregate -->|"estimates costs"| node_llm
node_aggregate -->|"loads defaults"| node_config
node_plots -->|"plots comparison"| node_aggregate
node_dashboard -->|"uses summaries"| node_aggregate
node_pkg_scores -->|"tokenizes programs"| node_pkg_tokenizer

click node_cli "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/__main__.py"
click node_config "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/config.py"
click node_orchestrator "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/orchestrator.py"
click node_planner "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/agents/planner.py"
click node_coder "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/agents/coder.py"
click node_llm "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/agents/llm_client.py"
click node_tester "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/agents/tester.py"
click node_humaneval "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/benchmarks/humaneval.py"
click node_mbpp "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/benchmarks/mbpp.py"
click node_sandbox "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/sandbox/docker_runner.py"
click node_critic "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/agents/critic.py"
click node_tokenizer "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/metrics/tokenizer.py"
click node_scores "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/metrics/scores.py"
click node_diagnosis "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/feedback/diagnosis.py"
click node_templates "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/feedback/templates.py"
click node_trivial "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/feedback/trivial.py"
click node_aggregate "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/reporting/aggregate.py"
click node_plots "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/reporting/plots.py"
click node_dashboard "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/dashboard.py"
click node_run_data "https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/orchestrator.py"
click node_pkg_tokenizer "https://github.com/yomileonel/beyondpass/blob/main/packages/beyondpass-metrics/src/beyondpass_metrics/tokenizer.py"
click node_pkg_scores "https://github.com/yomileonel/beyondpass/blob/main/packages/beyondpass-metrics/src/beyondpass_metrics/scores.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_config,node_orchestrator,node_planner,node_coder,node_llm toneBlue
class node_critic,node_tokenizer,node_scores,node_diagnosis,node_templates,node_trivial toneAmber
class node_tester,node_humaneval,node_mbpp,node_sandbox,node_openai toneMint
class node_aggregate,node_plots,node_dashboard,node_run_data toneRose
class node_pkg_tokenizer,node_pkg_scores,node_user toneIndigo
```

## Components

### Run workflow

- [Run and report CLI](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/__main__.py): `src/beyondpass/__main__.py`
- [Run configuration](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/config.py): `src/beyondpass/config.py`
- [Iteration orchestrator](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/orchestrator.py): `src/beyondpass/orchestrator.py`
- [Planner agent](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/agents/planner.py): `src/beyondpass/agents/planner.py`
- [Coder agent](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/agents/coder.py): `src/beyondpass/agents/coder.py`
- [LLM client](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/agents/llm_client.py): `src/beyondpass/agents/llm_client.py`

### Structural feedback

- [Critic agent](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/agents/critic.py): `src/beyondpass/agents/critic.py`
- [AST tokenizer](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/metrics/tokenizer.py): `src/beyondpass/metrics/tokenizer.py`
- [Structural scores](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/metrics/scores.py): `src/beyondpass/metrics/scores.py`
- [Failure diagnosis](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/feedback/diagnosis.py): `src/beyondpass/feedback/diagnosis.py`
- [Feedback templates](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/feedback/templates.py): `src/beyondpass/feedback/templates.py`
- [Trivial-solution flags](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/feedback/trivial.py): `src/beyondpass/feedback/trivial.py`

### Benchmarks and isolation

- [Tester agent](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/agents/tester.py): `src/beyondpass/agents/tester.py`
- [HumanEval loader](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/benchmarks/humaneval.py): `src/beyondpass/benchmarks/humaneval.py`
- [MBPP loader](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/benchmarks/mbpp.py): `src/beyondpass/benchmarks/mbpp.py`
- [Docker sandbox](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/sandbox/docker_runner.py): `src/beyondpass/sandbox/docker_runner.py`

### Reporting

- [Run aggregation](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/reporting/aggregate.py): `src/beyondpass/reporting/aggregate.py`
- [Comparison plots](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/reporting/plots.py): `src/beyondpass/reporting/plots.py`
- [Interactive dashboard](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/dashboard.py): `src/beyondpass/dashboard.py`
- [JSONL run results](https://github.com/yomileonel/beyondpass/blob/main/src/beyondpass/orchestrator.py): `src/beyondpass/orchestrator.py`

### Standalone metrics

- [Packaged tokenizer](https://github.com/yomileonel/beyondpass/blob/main/packages/beyondpass-metrics/src/beyondpass_metrics/tokenizer.py): `packages/beyondpass-metrics/src/beyondpass_metrics/tokenizer.py`
- [Packaged scores](https://github.com/yomileonel/beyondpass/blob/main/packages/beyondpass-metrics/src/beyondpass_metrics/scores.py): `packages/beyondpass-metrics/src/beyondpass_metrics/scores.py`

### Other components

- **Engineer**
- **OpenAI API**

## Connections

- Engineer → Run and report CLI: starts run (evidence: `README.md`)
- Run and report CLI → Run configuration: loads settings (evidence: `src/beyondpass/__main__.py`)
- Run and report CLI → Iteration orchestrator: starts evaluation (evidence: `src/beyondpass/__main__.py`)
- Run and report CLI → HumanEval loader: loads tasks (evidence: `src/beyondpass/__main__.py`)
- Run and report CLI → MBPP loader: loads tasks (evidence: `src/beyondpass/__main__.py`)
- Run and report CLI → Run aggregation: requests summary (evidence: `src/beyondpass/__main__.py`)
- Run and report CLI → Comparison plots: renders comparison (evidence: `src/beyondpass/__main__.py`)
- Iteration orchestrator → Planner agent: requests plan (evidence: `src/beyondpass/orchestrator.py`)
- Iteration orchestrator → Coder agent: requests candidate (evidence: `src/beyondpass/orchestrator.py`)
- Iteration orchestrator → Tester agent: tests candidate (evidence: `src/beyondpass/orchestrator.py`)
- Iteration orchestrator → Critic agent: scores candidate (evidence: `src/beyondpass/orchestrator.py`)
- Iteration orchestrator → Feedback templates: builds feedback (evidence: `src/beyondpass/orchestrator.py`)
- Iteration orchestrator → Run configuration: uses settings (evidence: `src/beyondpass/orchestrator.py`)
- Iteration orchestrator → JSONL run results: writes results (evidence: `src/beyondpass/orchestrator.py`)
- Planner agent → LLM client: requests completion (evidence: `src/beyondpass/agents/planner.py`)
- Coder agent → LLM client: requests completion (evidence: `src/beyondpass/agents/coder.py`)
- LLM client → OpenAI API: calls API (evidence: `src/beyondpass/agents/llm_client.py`)
- Tester agent → Docker sandbox: executes candidate (evidence: `src/beyondpass/agents/tester.py`)
- Critic agent → Structural scores: computes metrics (evidence: `src/beyondpass/agents/critic.py`)
- Critic agent → Failure diagnosis: classifies failure (evidence: `src/beyondpass/agents/critic.py`)
- Critic agent → Trivial-solution flags: checks input use (evidence: `src/beyondpass/agents/critic.py`)
- Structural scores → AST tokenizer: tokenizes programs (evidence: `src/beyondpass/metrics/scores.py`)
- Failure diagnosis → Run configuration: uses thresholds (evidence: `src/beyondpass/feedback/diagnosis.py`)
- Run aggregation → JSONL run results: reads results
- Run aggregation → LLM client: estimates costs (evidence: `src/beyondpass/reporting/aggregate.py`)
- Run aggregation → Run configuration: loads defaults (evidence: `src/beyondpass/reporting/aggregate.py`)
- Comparison plots → Run aggregation: plots comparison (evidence: `src/beyondpass/reporting/plots.py`)
- Interactive dashboard → Run aggregation: uses summaries (evidence: `src/beyondpass/dashboard.py`)
- Packaged scores → Packaged tokenizer: tokenizes programs (evidence: `packages/beyondpass-metrics/src/beyondpass_metrics/scores.py`)
