How md-ashraful-pramanik/mapcoder works
Main Components
• Entry Point/Controller: Represented by src/main.py, which parses command-line arguments (selecting model, dataset, and strategy) and drives the whole pipeline.
• Environment & Configuration: The .env files (.env.example.azure, .env.example.openai) and requirements.txt that configure external services and dependencies.
• Data & Datasets: The “data” directory holds various benchmark datasets (HumanEval, MBPP, APPS, CodeContest, xCodeEval) used for evaluation. Within src/datasets, you have dataset loaders (APPSDataset, CodeContestDataset, etc.) and a factory (DatasetFactory) that abstracts different dataset types.
• Agent Modules (Prompting): Found in src/promptings, including distinct files for Analogical, CoT (Chain-of-Thought), Direct, MapCoder, SelfPlanning. These files encapsulate the multi-agent strategies – with each file generally corresponding to a specific agent in the overall workflow.
• Model Management: Under src/models, you have classes for different underlying LLM models (OpenAI, Gemini, etc.) and a ModelFactory for configuration and instantiation.
• Evaluations and Execution: The src/evaluations package includes modules for interacting with external APIs (api_comm.py), evaluating generated code, and executing unit tests (exec_outcome.py, func_evaluate.py). Additionally, there is integration with an external dockerized service (ExecEval) for competitive dataset execution.
• Utility Components: The src/utils directory with utility modules (jsonl parsing, token counting, parsing) that assist both evaluation and data handling tasks.
• Constants: The src/constants repository contains shared constants like language mappings and paths.
Relationships and Data Flow
• The main.py acts as the central coordinator: it reads configuration, sets up the environment, and triggers the data processing, model selection, and evaluation pipelines.
• The dataset components (in src/datasets) load and convert dataset information (also with conversion scripts like convert-apps-xcode.py) and funnel test problems into the evaluation pipeline.
• The model components, selected via ModelFactory, interact with the various prompting strategies. The prompting modules (in src/promptings) are arranged in a pipeline that mimics a human-like coding cycle:
• The Retrieval Agent first recollects similar, relevant coding examples.
• The Planning Agent then produces a step-by-step solution plan.
• The Coding Agent converts this plan into actual code.
• Finally, the Debugging Agent revises and repairs the code based on test feedback.
• The evaluation package executes, validates, and records results (storing outputs into the “final-results” directory) across multiple benchmarks. The API communication module (api_comm.py) hints at additional external interactions.
• Utility functions support JSON data handling and token counting to ensure that prompt lengths and outputs meet specific constraints.
Architectural Patterns and Principles
• Modular Design: The separation of major concerns into discrete directories (datasets, models, promptings, evaluations) demonstrates a clean multi-layered modular architecture.
• Factory and Abstraction Patterns: Both the DatasetFactory and the ModelFactory encapsulate instantiation details for various datasets and models.
• Pipeline-Oriented Flow: The sequential flow of the multi-agent cycle (retrieval → planning → coding → debugging) is a central architectural concept.
• Extensibility: Clear separation into folders (e.g., different prompting strategies) makes it easy to add new agents or modify behavior without impacting unrelated parts of the system.
• Nodes/Components with clear, descriptive labels: “Main Controller (src/main.py)”, “Dataset Loader (src/datasets, conversion scripts)”, “Retrieval Agent”, “Planning Agent”, “Coding Agent”, “Debugging Agent”, “Model Factory (src/models)”, “Evaluation Engine (src/evaluations)”, “Utilities & Constants”, “External Services (ExecEval, APIs)”.
• Annotations for design patterns employed (factories, pipeline sequence).
• A clear demarcation between configuration files, data/input (from the “data” directory), processing modules, and output (saved in “final-results”).