How cmu-agents/assignment-2 works
This is an assignment toolkit for evaluating a data-visualization agent: tasks are authored and packaged, agents produce run artifacts, validators judge those runs, and human labels support scoring validator predictions. The graph separates task/run generation, validation, and evaluation while showing the model-serving integration. Sampled source confirms the validator and model calls; links involving unsampled workflow entry points and generation orchestration are omitted where their ownership cannot be verified.
Main entry points: artifacts Run Artifacts · runner.py Validation Runner · tasks Authored Tasks · generate.py Run Generation
Components 21
Tasks and Runs
- Authored Tasks
tasks - Task Validation
workflow/tasks.py - Task Packaging
workflow/package.py - Harbor Tasks
harbor - Run Generation
workflow/generate.py - Run Artifacts
artifacts
Agent and Serving
- Agent Execution
infrastructure/agent_runner.py - Generation Gateway
infrastructure/generation_gateway.py - Generation Models
infrastructure/generation_models.py
Validation
- Validation Runner
validator/runner.py - Validator Solution
validator/solution.py - Baseline Judge
validator/baseline.py - Judge Model Client
validator/model.py - Predictions
validator/prediction.py
Evaluation
- Labeling and Scoring
workflow/evaluate.py - Submission Checks
workflow/submission.py - Harbor Rewards
workflow/harbor_rewards.py
Other components
- Task Author
- Generation Endpoint
- Validator Endpoint
- Human Reviewer
Connections 21
- Task Author → Authored Tasks: authors
- Authored Tasks → Task Validation: loads
- Task Validation → Run Generation: provides tasks
- Run Generation → Agent Execution: dispatches runs
- Agent Execution → Generation Gateway: requests generation
- Generation Gateway → Generation Endpoint: forwards requests
- Run Generation → Run Artifacts: records results
- Authored Tasks → Task Packaging: packages
- Task Packaging → Harbor Tasks: creates tasks
- Validation Runner → Run Artifacts: loads artifacts
- Validation Runner → Validator Solution: invokes
- Validator Solution → Baseline Judge: uses judge
- Baseline Judge → Judge Model Client: asks questions
- Judge Model Client → Validator Endpoint: sends requests
- Validation Runner → Predictions: validates and writes
- Human Reviewer → Labeling and Scoring: reviews runs
- Labeling and Scoring → Run Artifacts: loads artifacts
- Labeling and Scoring → Predictions: scores
- Submission Checks → Harbor Tasks: checks packaging
- Submission Checks → Run Artifacts: checks records
- Harbor Rewards → Harbor Tasks: runs tasks