How kagnlp/codegenerator works
Main Components
• Data Inputs & Datasets:
• Datasets directory (data) holds multiple benchmark datasets (APPS, HumanEval, MBPPEval, xCodeEval). These are the source problems on which the system operates.
• Supporting dataset conversion scripts exist in src/datasets (e.g., convert-apps-xcode.py).
• Core Agents:
• Planning Agent (from src/promptings, particularly the “Base”, “CoT”, “SelfPlanning” variations, etc.) is responsible for analyzing the problem description and generating a plan with exemplar guidance.
• Coding Agent (from src/models and possibly tied with promptings) which takes the generated plan and transforms it into executable code.
• Debugging Agent (covered in src/evaluations and src/utils) which simulates the execution, analyzes test case failures, and refines the code.
• Execution & Evaluation:
• Evaluation and simulation modules are located in src/evaluations and src/utils. These components use sample input/output test cases (and possibly integrate with external services like ExecEval) for verifying generated code.
• There is an API communication component (api_comm.py) to handle external execution environments. This supports docker-based execution as noted in the README.
• Results:
• The resulting outputs and logs are handled by components in src/results.
• Configurations & Constants:
• The project uses configuration constants under src/constants and environment configuration via .env.example.
Relationships and Interactions
• Input Flow:
• The user provides a problem description (and selects a specific dataset via command-line arguments).
• Data from benchmark datasets is loaded and pre-processed by the dataset components.
• Processing Flow:
• The Planning Agent takes in the problem and generates a plan (using strategies like exemplar-based planning and simulation verification).
• The Coding Agent uses the validated plan to produce executable code.
• The Debugging Agent receives the code along with execution logs and then simulates the code to identify errors and refine it.
• Output Flow:
• Results are captured, logged, and stored in the results module.
• There is also an integration point for evaluation against standard benchmarks, logging performance metrics (pass@1 scores, etc.).
• External and Auxiliary Services:
• Integration with external debuggers or runtime environments can be depicted (e.g., ExecEval via docker, with configuration set by modifications in api_comm.py).
Architectural Patterns and Design Principles
• Multi-Agent Pipeline:
• Sequential collaboration between the Planning, Coding, and Debugging agents, mirroring a human-like step-by-step approach.
• Modularity:
• Each agent is modularized (with specific modules under src/models, src/promptings, src/evaluations) making the system extensible.
• Simulation-Driven Verification:
• The design highlights simulation-based internal debugging, an approach central to CodeSim’s performance.
Technologies, Frameworks, and Libraries
• Python is the primary language, with dependencies listed in requirements.txt.
• Integration with external model providers like OpenAI is apparent (via src/models/OpenAI.py).
• YAML configuration files (e.g., limits_by_lang.yaml) indicate configuration-driven execution limits.
• Any LLM-based processing is embedded within the multi-agent system (seen in both promptings and evaluations modules).