How jennyzzt/dgm works
Project Type and Purpose
• This is a command-line, self-improving AI system (“Darwin Gödel Machine”) written in Python.
• It orchestrates an open-ended evolution loop: an LLM-based agent proposes code patches to itself, those patches are sandbox-tested via Docker against two benchmarks (SWE-bench and Polyglot), and performance data feeds back to drive further improvements.
High-Level Components
A. DGM Controller
• DGM_outer.py: the main entry point. Coordinates the self-improvement loop.
• self_improve_step.py: encapsulates one iteration: request patch, apply patch, evaluate, log results.
B. LLM Agent
• coding_agent.py & coding_agent_polyglot.py: implement the logic for the initial agent in each domain.
• llm.py & llm_withtools.py: wrap calls to OpenAI/Anthropic APIs and expose tool-use abstractions.
• prompts/: modular prompt templates used by the agent (e.g., testrepo_prompt.py, diagnose_improvement_prompt.py).
• tools/: Python implementations of “tools” the agent can invoke in-prompt (e.g., bash.py, edit.py).
C. Sandbox & Evaluation
• Docker orchestration:
• polyglot/docker_build.py, polyglot/docker_utils.py, utils/docker_utils.py build and manage containers.
• Individual patch evaluations run in isolated containers to protect the host.
• Benchmark harnesses:
• swe_bench/harness.py and its ref_agent_results/ for SWE-bench tasks.
• polyglot/harness.py for Polyglot tasks.
• Logging & Data:
• initial/ and initial_polyglot/: baseline run logs, patch diffs, reports, and predictions.
• Predictions in JSON/MD plus Docker logs and eval scripts.
D. Analysis & Visualization
• analysis/: scripts (plot_comparison.py, plot_progress.py, visualize_archive.py) to process logs and generate charts.
E. Utilities
• utils/: shared helper code (common_utils.py, eval_utils.py, git_utils.py, evo_utils.py, swe_log_parsers.py).
• polyglot/: dataset preparation and evaluation orchestration (prepare_polyglot_dataset.py, run_evaluation.py).
F. Testing & CI
• tests/: Pytest tests for tool wrappers and end-to-end DGM functionality (test_swebench.py, test_bash_tool.py, etc.).
• pytest.ini, requirements.txt, Dockerfile for reproducible environment setup.
Component Interactions & Data Flow
• Control Flow (DGM_outer) → invokes self_improve_step
↳ calls LLM Agent → uses prompts & tools → returns patch.diff
↳ applies patch (via git_utils.py) to sandbox copy of the code
↳ spins up Docker containers (via docker_utils)
• Runs swe_bench/harness and polyglot/harness in separate containers
• Produces test coverage, performance metrics, logs, and reports
↳ collects results (using eval_utils, swe_log_parsers)
↳ writes logs/metrics to filesystem under output_dgm/ (and initial folders)
↳ loop repeats until termination condition
• Analysis scripts read these logs to generate final plots and archives.
Architectural Patterns & Design Principles
• Modular CLI services: clear separation between the DGM controller, LLM agent, evaluation harnesses, and analysis.
• Sandboxing with Docker: ensures safety when executing untrusted, model-generated code.
• Tool abstractions for prompting: agents see standardized “tools” to inspect files or run shell commands.
• Benchmark plugins: SWE-bench and Polyglot harnesses plug into the same evaluation interface, making the system extensible to new benchmarks.
• Data-driven loop: each iteration feeds results back into the next, embodying self-improving, evolutionary principles.
Technologies & Frameworks
• Python 3.8+; uses virtual environments.
• LLM APIs: OpenAI, Anthropic (via llm.py).
• Docker for isolation.
• Pytest for testing.
• Graphviz (optional) for analysis visuals.
Essential Elements
• Clear component names and boundaries
• External dependencies (LLM endpoints, Docker)
• Storage areas for logs, predictions, archives