How frontisai/enterpriseclawbench works
EnterpriseClawBench constructs benchmark tasks from enterprise-agent session traces, runs tasks through an optional sandbox, and evaluates the resulting run directories against artifact rules and semantic rubrics. Its principal workflow proceeds from trace inventory and extraction through task cleaning, enrichment, selection, packaging, and export; an agent then produces responses and artifacts that the evaluation pipeline scores and reports. The graph retains the documented construction stages and the separate run/evaluation branches. Stage ordering and capabilities are documented, but their internal calls were not sampled, so those links reflect the documented workflow rather than inspected stage implementations.
Main entry points: runner.py Sandbox runner · judge.py Artifact and rubric judge · open_sandbox_client.py Sandbox client · session_protocol.py Exported tasks
Components 31
Task construction
- Construction CLI
construction/enterprise_clawbench_construction/cli.py - Pipeline runner
construction/enterprise_clawbench_construction/construction_core/pipeline.py - Trace inventory
construction/enterprise_clawbench_construction/stages/stage_00_inventory.py - Turn extraction
construction/enterprise_clawbench_construction/stages/stage_01_extract_turns.py - Instance segmentation
construction/enterprise_clawbench_construction/stages/stage_02_segment_instances.py - Mechanical checks
construction/enterprise_clawbench_construction/stages/stage_03_mechanical_checks.py - Mechanical join
construction/enterprise_clawbench_construction/stages/stage_04_mechanical_join.py - Prompt rewriting
construction/enterprise_clawbench_construction/stages/stage_05_self_contained.py - Prompt rewrite
construction/enterprise_clawbench_construction/stages/stage_06_prompt_rewrite.py - Task taxonomy
construction/enterprise_clawbench_construction/stages/stage_07_taxonomy.py - Skill subclasses
construction/enterprise_clawbench_construction/stages/stage_08_skill_subclass.py - Deliverable analysis
construction/enterprise_clawbench_construction/stages/stage_09_deliverables.py - Rule generation
construction/enterprise_clawbench_construction/stages/stage_10_rules.py - Semantic rubric
construction/enterprise_clawbench_construction/stages/stage_11_semantic_rubric.py - Candidate selection
construction/enterprise_clawbench_construction/stages/stage_12_candidate_selection.py - Task pack writer
construction/enterprise_clawbench_construction/stages/stage_13_pack_writer.py - Pack validation
construction/enterprise_clawbench_construction/stages/stage_14_pack_validate.py - Task export
construction/enterprise_clawbench_construction/stages/stage_15_export.py
Agent execution
- Sandbox runner
sandbox/runner.py - Task and fixture IO
sandbox/task_io.py - Sandbox client
sandbox/open_sandbox_client.py
Run evaluation
- Evaluation CLI
evaluation/enterprise_clawbench_evaluation/cli.py - Artifact and rubric judge
evaluation/enterprise_clawbench_evaluation/judge.py - Evidence preparation
evaluation/enterprise_clawbench_evaluation/evidence.py - Judge model client
evaluation/enterprise_clawbench_evaluation/judge_client.py - Evaluation report
evaluation/enterprise_clawbench_evaluation/report.py
Run protocol
- Exported tasks
construction/enterprise_clawbench_construction/session_protocol.py - Evaluation run files
evaluation/enterprise_clawbench_evaluation/run_protocol.py
Other components
- Raw sessions
- Agent system
- OpenAI-compatible API
Connections 34
- Construction CLI → Pipeline runner: starts
- Raw sessions → Trace inventory: supplies traces
- Pipeline runner → Trace inventory: runs stages
- Trace inventory → Turn extraction: passes inventory
- Turn extraction → Instance segmentation: passes turns
- Instance segmentation → Mechanical checks: passes instances
- Mechanical checks → Mechanical join: passes checks
- Mechanical join → Prompt rewriting: passes joined tasks
- Prompt rewriting → Prompt rewrite: passes tasks
- Prompt rewrite → Task taxonomy: passes prompts
- Task taxonomy → Skill subclasses: passes taxonomy
- Skill subclasses → Deliverable analysis: passes skills
- Deliverable analysis → Rule generation: passes deliverables
- Rule generation → Semantic rubric: passes rules
- Semantic rubric → Candidate selection: passes rubrics
- Candidate selection → Task pack writer: passes candidates
- Task pack writer → Pack validation: passes task pack
- Pack validation → Task export: passes validated pack
- Task export → Exported tasks: writes tasks
- Exported tasks → Sandbox runner: provides tasks
- Sandbox runner → Task and fixture IO: prepares cases
- Sandbox runner → Sandbox client: dispatches work
- Task and fixture IO → Sandbox client: supplies fixture uploads
- Sandbox client → Agent system: runs agent
- Agent system → Sandbox client: returns outputs
- Sandbox runner → Evaluation run files: writes run files
- Evaluation run files → Evaluation CLI: provides run data
- Exported tasks → Evaluation CLI: provides task definitions
- Evaluation CLI → Artifact and rubric judge: invokes judging
- Artifact and rubric judge → Evidence preparation: prepares evidence
- Artifact and rubric judge → Judge model client: requests model judgment
- Judge model client → OpenAI-compatible API: sends prompts
- OpenAI-compatible API → Judge model client: returns judgments
- Artifact and rubric judge → Evaluation report: records scores