How bwz96sco/or_llm_agent works
Project type and purpose
• This is an AI-agent framework (open-source tool) that orchestrates multiple LLM-based sub-agents to convert natural-language OR problems into mathematical models, generate solver code, debug it, execute via Gurobi, and evaluate results.
File structure
• Top-level scripts:
– or_llm_eval.py / or_llm_eval_async_resilient.py / or_llm_show.py: evaluation entry-points
– mcp_server.py: optional local microservice for multi-choice problems (MCP)
• Data layer (data/): dataset JSONs, processing scripts, conversion utilities
• Configuration: .env.example, python-version, requirements.txt, pyproject.toml
• Utilities: utils.py for shared helper functions
README for architecture insights
• Three sequential stages (sub-agents):
a. Mathematical Modeling Agent
b. Code Generation Agent
c. Debugging Agent
• Orchestrator runs each in turn, using LLM APIs (OpenAI, Claude, DeepSeek)
• Uses Gurobi Optimizer to solve generated code
• Batch and asynchronous evaluation supported
a. User / Client: supplies natural-language problem descriptions or dataset
b. Orchestrator (scripts/orchestrator module)
– Parses CLI args
– Loads problem input
– Invokes sub-agents sequentially or in parallel (async)
c. Modeling Agent
– Calls LLM API to produce a mathematical model (e.g., in Pyomo or Gurobi API)
d. Code Generation Agent
– Takes the model spec, prompts LLM to generate executable Python code
e. Debugging Agent
– Submits generated code to LLM for fix suggestions on errors
f. Execution Engine
– Runs the final code locally
– Interfaces with Gurobi Solver
g. Data Store
– Input datasets (data/datasets/*.json)
– Processed intermediate files
h. Evaluation & Reporting
– or_llm_eval*.py collects solutions, compares against ground truth, produces metrics
i. External Services
– LLM Providers (OpenAI, Claude, DeepSeek via API_BASE)
– Gurobi Optimizer
Relationships and interactions
• User → Orchestrator: CLI invocation or script call
• Orchestrator → Modeling Agent → LLM API → returns model spec
• Modeling Agent → Code Generation Agent → LLM API → returns code
• Code Generation Agent → Debugging Agent (if enabled) → LLM API → returns patched code
• Debugged code → Execution Engine → Gurobi → solution
• Solution → Evaluation module → metrics / logs → User
• Data Store is read by Orchestrator and agents; results stored or logged
Architectural patterns & design principles
• Pipeline / Agent Pattern: each sub-agent is a self-contained component with a single responsibility
• Prompts as Interfaces: LLMs abstracted as “services” behind prompt wrappers
• Async / Batch Processing: evaluation scripts support concurrency for throughput
• Modular Data Processing: scripts under data/ handle conversion, indexing, and metrics
Technologies & frameworks
• Python 3.8+
• LLM APIs: OpenAI (text-davinci family), Claude, DeepSeek
• Gurobi Python API (solver)
• Pytest or custom tests under data/ for dataset preparation
• Shell scripts (run_eval_batch_agent.sh) for orchestration