How hinthornw/trustcall works
Project Type and Purpose
• This is a Python library/package (trustcall) designed to improve reliability and efficiency when LLMs generate or update structured JSON outputs.
• It integrates with LLMs (OpenAI, LangChain‐Fireworks, etc.) and pydantic (or user‐defined Python functions) to iteratively generate JSON patches rather than full documents, handling validation errors gracefully.
File Structure Overview
• Top‐level:
• .github/workflows/ – CI pipelines for tests and evals.
• pyproject.toml – packaging and dependencies.
• Makefile – shortcuts (make test, make evals).
• _static/ – images used in README.
• Core code under trustcall/:
• _base.py – core extractor logic (prompt LLM, apply patches, retry).
• _validation_node.py – LangChain integration node for tool calling.
• py.typed – typed package.
• Tests under tests/:
• Unit tests (tests/unit_tests/) for utils, extraction, strict validation.
• Eval tests (tests/evals/) with VCR cassettes, benchmarking LLM performance.
Key Components
a. User Application – Your code that calls create_extractor(...) or llm.with_structured_output(...).
b. Trustcall Library –
• Extractor Core (_base.py): orchestrates prompts, applies patches, handles retries.
• Validation Node (_validation_node.py): wraps LangChain tool‐calling interface.
c. LLM Providers – OpenAI, ChatFireworks, or any LangChain‐compatible model endpoint.
d. Schema Validators – pydantic models or user functions defining JSON schema.
e. JSON Patch Engine – RFC6902 patch creation and application.
f. Test Harness & CI – pytest suite, VCR cassette playback, GitHub Actions workflows.
Relationships and Interactions
• User App → Trustcall Extractor: code invokes the extractor with schema/tool definitions and (optionally) existing JSON.
• Extractor → LLM Endpoint: sends prompt for initial JSON or patch operations.
• LLM → Extractor: returns JSON or patch operations.
• Extractor → Schema Validator: validates output; on errors, generates targeted JSON patch prompt.
• Extractor → JSON Patch Engine: applies patches to build up or update the object.
• Extractor → LLM (retry): repeats until schema validation passes or retry limit reached.
• Tests/CI → Repo: run unit tests, evals, and ensure code & LLM workflows behave as expected.
Architectural Patterns & Principles
• Iterative Refinement: generate small patches rather than full replacements.
• Separation of Concerns: core extraction logic is separate from LLM integration (_validation_node.py).
• Plugin‐style Tools: schemas can be pydantic models, Python functions, or JSON Schemas.
• Resilience: automatic retries on validation errors; cassette‐based testing for reproducibility.