How nerdai/llm-agents-from-scratch works
1. What this repo is
- llm_agents_from_scratch is a Python library that implements an educational LLM-agent stack from first principles.
- The codebase is organized around a small core runtime plus pluggable adapters for models, tools, skills, and memory.
2. Top-level runtime boundary
- The main runtime lives under src/llm_agents_from_scratch.
- Everything outside src is supporting material: notebooks, docs, capstones, talks, tests, and a few deployment/demo assets.
3. Agent orchestration layer
- agent/ is the central coordination layer.
- LLMAgentBuilder assembles an agent from an LLM backend, tool set, memory, and optional skill support.
- llm_agent.py is the execution path where user input is turned into prompts, tool calls are handled, and final responses are returned.
4. Core control flow
- Typical flow is: user prompt → agent → LLM → tool selection/output → tool execution → tool results back to LLM → final answer.
- The repo is structured to make that loop explicit rather than hiding it behind a large framework.
5. Model/provider abstraction
- base/llm.py defines the common LLM interface.
- llms/openai/ and llms/ollama/ are concrete provider adapters.
- This creates a clear boundary between agent logic and the external inference runtime, whether that is OpenAI’s API or a local Ollama server.
6. Tooling subsystem
- base/tool.py defines the tool contract.
- tools/simple_function.py and tools/pydantic_function.py adapt ordinary Python callables into LLM-callable tools.
- tools/default/ contains built-in utilities such as file reading and an interpreter-like tool.
- tools/mcp/ integrates Model Context Protocol tools through a provider and tool wrapper.
7. Skill subsystem
- skills/ supports filesystem-discovered “skills” stored as directories with SKILL.md, optional scripts, and resources.
- Discovery and validation live in skills/discovery.py, skills/skill.py, and skills/utils.py.
- Skills are a higher-level extension mechanism than tools: they package instructions, assets, and helper scripts that the agent can consume.
8. Memory subsystem
- memory/ provides the abstraction used by the agent to persist or recall conversational/contextual state.
- memory_stores/json.py offers a lightweight local store.
- memory_stores/qdrant/ connects to Qdrant for vector-backed retrieval and similarity-style memory.
- memory/recipes.py suggests higher-level memory patterns built on top of the store abstraction.
9. Shared data models and errors
- data_structures/ holds the canonical objects passed across subsystem boundaries: agent, LLM, memory, skill, and tool models.
- errors/ centralizes typed exceptions for agent, memory, MCP, skill, and task handling failures.
- These two packages act as the contract layer for the rest of the codebase.
10. Educational assets and examples
- examples/, more-examples/, docs/more-examples/, and capstones/ show the runtime in progressively richer scenarios.
- The notebooks are not just demos; they document how the agent, tool, skill, and memory subsystems compose in practice.
- extra/mcp-hailstone/ is a separate MCP server example that demonstrates external tool integration.
11. Documentation and publishing
- docs/ is a MkDocs site with API references, chapter notebooks, capstones, and rendered assets.
- github/workflows contains CI for linting, tests, docs publishing, and release automation.
- The docs are part of the repo’s architecture because they mirror the library’s module boundaries and usage patterns.
12. Testing and verification
- tests/ mirrors the source tree almost one-to-one, which makes the package boundaries explicit.
- There are dedicated tests for agent flow, LLM adapters, tools, skills, memory stores, notebook utilities, and API import stability.
- This indicates the architecture is intentionally modular and contract-driven.
13. Deployment/runtime extras
- docker/runpod/ provides a GPU-friendly container entrypoint for notebook or demo execution.
- The README’s default runtime assumes Ollama locally, with Lightning AI and Runpod as supported execution environments for larger models.
14. How to draw the architecture
- A good diagram would place agent/ in the center, fed by llms/, tools/, skills/, and memory/.
- Surround that with data_structures/ and errors/ as shared contracts.
- Show external boundaries for Ollama/OpenAI, Qdrant, and MCP servers, plus separate documentation/example environments around the core library.
Main entry points: llm_agents_from_scratch Library · tests Tests · memory Memory · examples Examples
Components 31
Core runtime
Main library under src/llm_agents_from_scratch.
- Library · Top-level Python package for the runtime and adapters.
src/llm_agents_from_scratch - Agent · Builds and runs the agent loop.
src/llm_agents_from_scratch/agent - Builder · Composes an agent from model, tools, memory, and skills.
src/llm_agents_from_scratch/agent/builder.py - Runtime loop · Turns user input into prompts, tool calls, and final responses.
src/llm_agents_from_scratch/agent/llm_agent.py - Base APIs · Abstract interfaces for models, tools, and memory stores.
src/llm_agents_from_scratch/base - LLM API · Common interface for LLM backends.
src/llm_agents_from_scratch/base/llm.py - Tool API · Common interface for callable tools.
src/llm_agents_from_scratch/base/tool.py - Store API · Common interface for persistence and retrieval backends.
src/llm_agents_from_scratch/base/memory_store.py - LLM adapters · Concrete providers that adapt external model runtimes.
src/llm_agents_from_scratch/llms - OpenAI · OpenAI chat and structured-output adapter.
src/llm_agents_from_scratch/llms/openai/llm.py - Ollama · Local Ollama adapter for model inference.
src/llm_agents_from_scratch/llms/ollama/llm.py - Tools · Tool adapters and built-in utilities used by the agent.
src/llm_agents_from_scratch/tools - Function tool · Wraps ordinary Python callables as LLM-callable tools.
src/llm_agents_from_scratch/tools/simple_function.py - Pydantic tool · Callable tool adapter with schema validation.
src/llm_agents_from_scratch/tools/pydantic_function.py - MCP tool · Tool wrapper for Model Context Protocol integration.
src/llm_agents_from_scratch/tools/mcp/tool.py - Skills · Filesystem-discovered skill packages with instructions, scripts, and resources.
src/llm_agents_from_scratch/skills - Memory · Conversation and context memory abstractions plus recipes.
src/llm_agents_from_scratch/memory - JSON store · Lightweight local memory store.
src/llm_agents_from_scratch/memory_stores/json.py - Qdrant store · Vector-backed memory store for retrieval-style memory.
src/llm_agents_from_scratch/memory_stores/qdrant/store.py - Models & errors · Shared data structures and typed exceptions used across subsystem boundaries.
src/llm_agents_from_scratch/data_structures
Examples and docs
Educational materials, notebooks, and published docs that exercise the core runtime.
- Examples · Educational notebooks that demonstrate the runtime in practice.
examples - Docs site · MkDocs site mirroring the library structure and usage patterns.
docs - Capstones · Long-form notebook projects and applied scenarios.
capstones - MCP demo · Standalone MCP server example used to demonstrate external tool integration.
extra/mcp-hailstone
Tests and CI
Automated verification, packaging, and release support.
- Tests · Package-level tests that mirror the source tree and verify contracts.
tests - CI workflows · Linting, testing, docs publishing, and release automation.
.github/workflows - Runpod image · Container entrypoint for GPU-friendly notebook or demo execution.
docker/runpod
External systems
Outside services and runtimes the library integrates with.
- OpenAI API · Hosted model inference runtime.
- Ollama server · Local model inference runtime.
- Qdrant · Vector database used by the memory store.
- MCP server · External Model Context Protocol tool provider.
Connections 47
- Library → Agent: contains · Core library entry point includes agent orchestration.
- Library → Base APIs: contains · Abstract interfaces live in the base layer.
- Library → LLM adapters: contains · Provider adapters live under llms.
- Library → Tools: contains · Tool adapters and built-ins live under tools.
- Library → Skills: contains · Skill discovery and validation live under skills.
- Library → Memory: contains · Memory abstractions and recipes live under memory.
- Library → Models & errors: contains · Shared models and typed errors are part of the contract layer.
- Agent → Builder: assembles · Builder wires dependencies into an agent instance.
- Agent → Runtime loop: executes · Runtime loop handles prompt, tool, and response flow.
- Builder → LLM API: uses · Builder depends on the model abstraction.
- Builder → Tool API: uses · Builder accepts tool implementations.
- Builder → Store API: uses · Builder can attach memory backends.
- Builder → Skills: enables · Builder can opt into skill support.
- Runtime loop → LLM API: calls · Agent loop asks the model for next actions or final output.
- Runtime loop → Tool API: invokes · Tool calls are executed during the agent loop.
- Runtime loop → Memory: reads/writes · Conversation state can be persisted and recalled.
- Runtime loop → Skills: consumes · Skills can augment prompts and capabilities.
- LLM adapters → OpenAI: implements · OpenAI provider adapter.
- LLM adapters → Ollama: implements · Ollama provider adapter.
- OpenAI → OpenAI API: connects to · Uses the hosted OpenAI service.
- Ollama → Ollama server: connects to · Uses a local Ollama server.
- Tools → Function tool: contains · Function-based tool adapter.
- Tools → Pydantic tool: contains · Validated callable tool adapter.
- Tools → MCP tool: contains · MCP integration lives in the tool layer.
- MCP tool → MCP server: connects to · Delegates tool execution to an external MCP provider.
- Skills → Models & errors: shares models · Skill definitions and errors use shared contracts.
- Memory → JSON store: uses · Local persistent store implementation.
- Memory → Qdrant store: uses · Vector-backed store implementation.
- Qdrant store → Qdrant: connects to · Persists and queries memory in Qdrant.
- Memory → Models & errors: shares models · Memory state and exceptions use shared contracts.
- Examples → Agent: demonstrates · Notebooks exercise the core agent flow.
- Examples → Tools: demonstrates · Examples show tool creation and execution.
- Examples → Skills: demonstrates · Examples show filesystem skills in action.
- Examples → Memory: demonstrates · Examples include memory patterns and retrieval.
- Docs site → Library: documents · Docs mirror the runtime and API boundaries.
- Docs site → Examples: publishes · Docs include notebook-based examples and chapters.
- Capstones → Agent: applies · Capstone notebooks build on the agent runtime.
- MCP demo → MCP server: implements · Standalone example MCP server.
- Tests → Agent: verifies · Agent flow and builder tests.
- Tests → LLM adapters: verifies · Adapter tests for model providers.
- Tests → Tools: verifies · Tool adapters and MCP tests.
- Tests → Skills: verifies · Skill discovery and validation tests.
- Tests → Memory: verifies · Store and recipes tests.
- Tests → Models & errors: verifies · API stability and model contract tests.
- CI workflows → Tests: runs · Workflows execute the test suite.
- CI workflows → Docs site: publishes · Workflows build and publish documentation.
- Runpod image → Examples: hosts · Container supports notebook/demo execution.