How guidance-ai/guidance works
Based on our analysis of both the file tree and the README, Guidance is a Python library that provides a programming paradigm to “steer” language models. It does so by letting developers write “guidance functions” (decorated Python functions) that interleave static text with controlled generation calls. The project supports multiple backends (Transformers, llama.cpp, OpenAI, Vertex AI, etc.), constrained generation (through regex, CFGs, selects), and even tool integration (for automatic calls like a calculator). There is also a client-side component (implemented in Svelte) and a comprehensive set of notebooks, tests, and even packaging for web widgets (in the “stitch” package).
Major Components
• Core Guidance Library: This is the heart of the project and resides in the “guidance” directory. Its submodules include components for parsing and executing guidance programs (_ast, _grammar, _parser, _schema, _guidance, _utils), metrics gathering, visualizations (trace, visual), and integration with various models (e.g. models, registry, library).
• Tooling / Domain-specific Grammars & Extensions: Submodules in “guidance/library” (audio, image, json, etc.), along with the “stitch” package under “packages/python/stitch,” provide encapsulated reusable functionalities and possibly UI widgets.
• Client Interface: Located in “client/graphpaper-inline”, which appears to be a Svelte-based frontend for interactive use and demonstration of guidance functionality.
• Documentation and Notebooks: The “docs” and “notebooks” directories provide examples, tutorials, and API documentation that illustrate how the library is used for constrained generation, stateful control, and tool interleaving.
• Testing & CI: There is a comprehensive testing suite (in “tests”) and several CI configurations (in “.github/workflows”) that ensure proper functioning across various components.
Relationships and Interactions
• The Core Guidance Library sits at the center and orchestrates generation logic. It uses control structures (e.g., the @guidance decorator) to build “lm” objects that capture prompt state.
• Tools and Callables: Mark a subsystem (or set of components) within the library where tools (like the calculator tool) are defined and integrated seamlessly via automatic interruption and resumption of generation.
• Documentation, Notebooks, and Tests should be represented as supporting artifacts that either feed into education or validate the product, without being “runtime” components.
Architectural Patterns and Design Principles
• Modular and Pluggable Architecture: Emphasize that the project is broken down into independent modules (guidance/_* modules, models, library) that are loosely coupled. This supports multiple backends and custom tooling.
• Separation of Concerns: The frontend client, core generation logic, model integrations, and extension libraries are implemented in separate directories, each with well-defined roles.
• Stateful Control: Highlight the state management approach (immutable “lm” objects) that allows fine-grained control of text generation and interleaving of generated text with static text.
• Tool Invocation Mechanism: Illustrate the design pattern where generation is interrupted for tool calls (using specific grammars) and then resumed, ensuring that control and generation are interleaved seamlessly.
– Python for the core library and scripting
– JavaScript/Svelte (in client/graphpaper-inline) for the user interface
– CI/CD scripts (GitHub Actions workflows)
– Documentation using reStructuredText (with ReadTheDocs)
Specific Elements
– Parsing & Grammar ([_ast, _grammar, _parser, _schema])
– Control & Generation (_guidance, gen, select, capture)
– Tracing & Visualizations (trace, visual)
– Different adapters (OpenAI, LlamaCpp, Transformers, Vertex AI) coming off the main guidance module.
– Showcase the frontend “graphpaper-inline” client.
– Highlight modules like the calculator tool, and grammars for constrained generation (select, regex, CFGs).