How yomileonel/beyondpass works
BeyondPass evaluates LLM-generated solutions on HumanEval or MBPP tasks through a bounded Planner–Coder–Tester–Critic loop. The Critic compares candidates with references using AST-based structural scores, diagnoses failures, and supplies feedback for retries; the run also records results for comparison and visualization. The graph includes the standalone metrics package and optional dashboard. Some links rely on README or SOURCE INDEX references rather than inspected implementation excerpts, so unsampled internals are not expanded.
Main entry points: orchestrator.py Iteration orchestrator · __main__.py Run and report CLI · aggregate.py Run aggregation · critic.py Critic agent
Components 24
Run workflow
- Run and report CLI
src/beyondpass/__main__.py - Run configuration
src/beyondpass/config.py - Iteration orchestrator
src/beyondpass/orchestrator.py - Planner agent
src/beyondpass/agents/planner.py - Coder agent
src/beyondpass/agents/coder.py - LLM client
src/beyondpass/agents/llm_client.py
Structural feedback
- Critic agent
src/beyondpass/agents/critic.py - AST tokenizer
src/beyondpass/metrics/tokenizer.py - Structural scores
src/beyondpass/metrics/scores.py - Failure diagnosis
src/beyondpass/feedback/diagnosis.py - Feedback templates
src/beyondpass/feedback/templates.py - Trivial-solution flags
src/beyondpass/feedback/trivial.py
Benchmarks and isolation
- Tester agent
src/beyondpass/agents/tester.py - HumanEval loader
src/beyondpass/benchmarks/humaneval.py - MBPP loader
src/beyondpass/benchmarks/mbpp.py - Docker sandbox
src/beyondpass/sandbox/docker_runner.py
Reporting
- Run aggregation
src/beyondpass/reporting/aggregate.py - Comparison plots
src/beyondpass/reporting/plots.py - Interactive dashboard
src/beyondpass/dashboard.py - JSONL run results
src/beyondpass/orchestrator.py
Standalone metrics
- Packaged tokenizer
packages/beyondpass-metrics/src/beyondpass_metrics/tokenizer.py - Packaged scores
packages/beyondpass-metrics/src/beyondpass_metrics/scores.py
Other components
- Engineer
- OpenAI API
Connections 29
- Engineer → Run and report CLI: starts run
README.md - Run and report CLI → Run configuration: loads settings
src/beyondpass/__main__.py - Run and report CLI → Iteration orchestrator: starts evaluation
src/beyondpass/__main__.py - Run and report CLI → HumanEval loader: loads tasks
src/beyondpass/__main__.py - Run and report CLI → MBPP loader: loads tasks
src/beyondpass/__main__.py - Run and report CLI → Run aggregation: requests summary
src/beyondpass/__main__.py - Run and report CLI → Comparison plots: renders comparison
src/beyondpass/__main__.py - Iteration orchestrator → Planner agent: requests plan
src/beyondpass/orchestrator.py - Iteration orchestrator → Coder agent: requests candidate
src/beyondpass/orchestrator.py - Iteration orchestrator → Tester agent: tests candidate
src/beyondpass/orchestrator.py - Iteration orchestrator → Critic agent: scores candidate
src/beyondpass/orchestrator.py - Iteration orchestrator → Feedback templates: builds feedback
src/beyondpass/orchestrator.py - Iteration orchestrator → Run configuration: uses settings
src/beyondpass/orchestrator.py - Iteration orchestrator → JSONL run results: writes results
src/beyondpass/orchestrator.py - Planner agent → LLM client: requests completion
src/beyondpass/agents/planner.py - Coder agent → LLM client: requests completion
src/beyondpass/agents/coder.py - LLM client → OpenAI API: calls API
src/beyondpass/agents/llm_client.py - Tester agent → Docker sandbox: executes candidate
src/beyondpass/agents/tester.py - Critic agent → Structural scores: computes metrics
src/beyondpass/agents/critic.py - Critic agent → Failure diagnosis: classifies failure
src/beyondpass/agents/critic.py - Critic agent → Trivial-solution flags: checks input use
src/beyondpass/agents/critic.py - Structural scores → AST tokenizer: tokenizes programs
src/beyondpass/metrics/scores.py - Failure diagnosis → Run configuration: uses thresholds
src/beyondpass/feedback/diagnosis.py - Run aggregation → JSONL run results: reads results
- Run aggregation → LLM client: estimates costs
src/beyondpass/reporting/aggregate.py - Run aggregation → Run configuration: loads defaults
src/beyondpass/reporting/aggregate.py - Comparison plots → Run aggregation: plots comparison
src/beyondpass/reporting/plots.py - Interactive dashboard → Run aggregation: uses summaries
src/beyondpass/dashboard.py - Packaged scores → Packaged tokenizer: tokenizes programs
packages/beyondpass-metrics/src/beyondpass_metrics/scores.py