How antoinezambelli/forge works
1. What this repository is
Forge is a Python 3.12+ package that sits between an agent loop and an LLM backend to make multi-step tool use more reliable. Its architecture centers on a shared inference pipeline that can be used directly in application code, embedded as middleware, or exposed as an OpenAI-compatible proxy.
2. Top-level execution paths
There are three primary entry points:
- WorkflowRunner for end-to-end agent workflows managed by Forge.
- Guardrails middleware for projects that already own the loop but want Forge’s validation and recovery.
- Proxy server for drop-in use behind OpenAI-compatible clients.
These three paths share the same core inference, guardrail, context, and client layers.
3. Core orchestration layer
forge.core is the center of the system. It defines the workflow model, message types, inference pipeline, step tracking, and the main agent loop. run_inference() is the shared “front half” that prepares context, validates/repairs responses, and hands back structured model output. WorkflowRunner builds the full cycle around that inference path, including tool execution and continuation.
4. Workflow and tool model
Workflows are declarative: they define a name, description, tool set, required steps, a terminal tool, and a system prompt template. Each tool is represented by a ToolDef plus a ToolSpec, and tool calls are turned into structured ToolCall or TextResponse outputs. This is the boundary where model output becomes executable application behavior.
5. Backend client abstraction
forge.clients isolates model-provider differences behind a common LLM client protocol. The repo supports:
- Ollama with native function calling,
- llama-server / llama.cpp with native function calling or prompt-injected mode,
- Llamafile as a single-binary backend,
- Anthropic as a hosted baseline.
This layer is the outbound boundary from Forge to inference infrastructure.
6. Context management subsystem
forge.context manages token budgets and compaction. ContextManager applies a chosen strategy such as sliding window or tiered compaction, while hardware detection informs budgeting decisions. This subsystem is critical because the repo is optimized for constrained local models and needs to preserve the important parts of multi-turn tool interactions within VRAM-aware limits.
7. Guardrails and recovery
forge.guardrails is the reliability layer. It includes response validation, retry nudges, step enforcement, and error tracking. The guardrails are designed to catch malformed tool responses, enforce required workflow steps, and recover from common failure modes without exposing that complexity to the caller.
8. Prompting and synthetic tools
forge.prompts contains templates for tool-injected prompting and for retry/step-enforcement nudges. forge.tools.respond provides the synthetic respond tool used by the proxy when text output must be coerced into tool-calling mode. This is a key architectural trick: the model is guided to emit structured intent, then Forge strips the synthetic tool from the final client-visible response.
9. Proxy boundary
forge.proxy exposes an asyncio-based HTTP server that speaks OpenAI-compatible APIs. It converts incoming OpenAI-style messages into Forge messages, runs them through the same inference/guardrail pipeline, and streams responses back. The proxy can run in external mode against an already-running backend or managed mode where Forge launches the backend alongside the proxy.
10. Shared runtime / server management
forge.server and related runtime helpers coordinate backend startup and budget mode selection. This layer is what lets the package manage local model servers in a predictable way, especially for proxy-managed deployments and workflows that need backend lifecycle control.
11. SlotWorker and multi-agent access
forge.core.slot_worker adds a priority-queued access model to a shared inference slot. This is a boundary for multi-agent or multi-workflow systems that need one GPU-backed model instance shared across competing tasks, with preemption and scheduling handled centrally instead of by the caller.
12. Evaluation and validation infrastructure
The tests/eval tree is a full evaluation harness, not just unit tests. It includes scenario definitions, batch evaluation, metrics, significance testing, and an HTML dashboard for reporting. This infrastructure is important architecturally because the project’s claims and design decisions are grounded in a 26-scenario eval suite, with separate baselines and reasoning-oriented tiers.
13. Repository boundaries
tests/unit validates deterministic behavior for the core subsystems without requiring a live model backend. tests/eval exercises real backend integrations and model behavior. Documentation under docs/ captures architecture decisions, workflow internals, model/backend setup, and published evaluation results; it functions as the design record for the system rather than just user docs.
14. Architecture summary for a diagram
The clearest diagram is: application or proxy client on the left; Forge core in the middle; guardrails, context management, and workflow orchestration as the central control plane; client adapters on the outbound side; and one of several backends on the far right. Tool execution loops back from the workflow layer into the application-defined tools, while eval/test infrastructure sits alongside the runtime as a validation and regression boundary.
Main entry points: base.py Client protocol · inference.py Inference · guardrails.py Guardrails · runner.py WorkflowRunner
Components 28
Core runtime
Workflow execution and shared inference path
- WorkflowRunner · End-to-end workflow execution and continuation loop
src/forge/core/runner.py - Inference · Shared front-half that prepares context and model requests
src/forge/core/inference.py - Workflow model · Declarative workflow, tools, and terminal-step definitions
src/forge/core/workflow.py - Messages · Core message and response representations
src/forge/core/messages.py - Steps · Step tracking and progression rules
src/forge/core/steps.py - Slot worker · Priority-queued shared inference access
src/forge/core/slot_worker.py
I/O boundaries
Proxy and backend client interfaces
- Client protocol · Common LLM client boundary
src/forge/clients/base.py - Ollama · Native function-calling client
src/forge/clients/ollama.py - Llama server · llama.cpp or llama-server client modes
src/forge/clients/base.py - Llamafile · Single-binary backend client
src/forge/clients/llamafile.py - Anthropic · Hosted baseline client
src/forge/clients/anthropic.py - Proxy server · OpenAI-compatible asyncio proxy
src/forge/proxy/server.py - Convert · OpenAI message translation layer
src/forge/proxy/convert.py - Handler · Proxy request execution and streaming
src/forge/proxy/handler.py - Server mgmt · Backend startup and lifecycle coordination
src/forge/server.py - Respond tool · Synthetic tool used to coerce text output
src/forge/tools/respond.py
Reliability
Context control and guardrails
- Context manager · Token-budget and compaction coordinator
src/forge/context/manager.py - Strategies · Sliding-window and tiered context policies
src/forge/context/strategies.py - Hardware · Hardware-aware budget inputs
src/forge/context/hardware.py - Guardrails · Validation, retry, and failure recovery coordinator
src/forge/guardrails/guardrails.py - Validator · Checks model responses and tool output shape
src/forge/guardrails/response_validator.py - Step enforcer · Requires workflow step completion and order
src/forge/guardrails/step_enforcer.py - Nudges · Retry prompts and corrective hints
src/forge/guardrails/nudge.py - Prompts · System templates and retry prompt text
src/forge/prompts/templates.py
Validation
Evaluation and regression harness
- Eval suite · Scenario-based regression and benchmark harness
tests/eval/eval_runner.py - Dashboard · HTML/TSX results viewer for eval runs
tests/eval/dashboard/src/App.tsx
Docs
Architecture and published guidance
- Architecture doc · System architecture reference
docs/ARCHITECTURE.md - Eval guide · Evaluation workflow and reporting guidance
docs/EVAL_GUIDE.md
Connections 29
- WorkflowRunner → Inference: calls · Workflow execution begins with shared inference
- WorkflowRunner → Workflow model: uses · Workflow definitions drive execution
- WorkflowRunner → Steps: tracks · Advances and records workflow steps
- Inference → Context manager: requests context · Inference prepares token budget and compaction
- Inference → Guardrails: validates · Model output is checked before returning
- Inference → Client protocol: dispatches via · Inference calls the client abstraction
- Inference → Messages: shapes · Uses shared message and response types
- Context manager → Strategies: selects · Applies a compaction strategy
- Context manager → Hardware: adapts to · Budgeting reflects available hardware
- Guardrails → Validator: runs · Checks for malformed or incomplete responses
- Guardrails → Step enforcer: enforces · Ensures required workflow steps
- Guardrails → Nudges: repairs with · Retries are guided by corrective nudges
- Prompts → Nudges: supplies · Prompt templates back retry guidance
- Client protocol → Ollama: adapts · Concrete Ollama backend integration
- Client protocol → Llama server: adapts · llama-server / llama.cpp integration
- Client protocol → Llamafile: adapts · Llamafile integration
- Client protocol → Anthropic: adapts · Anthropic integration
- Proxy server → Convert: translates · Converts OpenAI-style payloads
- Proxy server → Handler: routes · Handles request execution and streaming
- Handler → Inference: reuses · Proxy shares the same inference path
- Handler → Respond tool: injects · Synthetic respond tool supports text coercion
- Respond tool → Guardrails: finalized by · Synthetic tool output is stripped or normalized
- Server mgmt → Client protocol: launches backends · Runtime management coordinates model servers
- Server mgmt → Proxy server: serves · Proxy-managed deployments use server lifecycle control
- Eval suite → WorkflowRunner: benchmarks · Evaluation exercises workflow execution
- Eval suite → Client protocol: measures · Eval runs against real backend clients
- Dashboard → Eval suite: renders · Dashboard visualizes evaluation outputs
- Architecture doc → WorkflowRunner: documents · Architecture doc describes the core runtime
- Eval guide → Eval suite: documents · Eval guide explains the harness and reporting