How langchain-ai/openevals works
• Defining evaluators that check output correctness, conciseness, hallucination, and various code correctness metrics.
• Integrating with external LLM providers (e.g. OpenAI, Anthropic) via LangChain, and with sandboxed execution environments using E2B.
• Providing support for multiple evaluation strategies including exact match, string similarity (e.g., Levenshtein, embedding similarity) and specialized code evaluators (e.g. pyright, mypy, TypeScript type-checking).
• Incorporating logging and experiment tracking via LangSmith.
Major Components and Layers
• Evaluators Module (Core Library):
– LLM-as-Judge Evaluators: These use prebuilt prompts (e.g. for correctness, conciseness, hallucination) that format inputs/outputs, invoke an external LLM (provided by LangChain integrations), and parse the resulting score.
– Structured Output & JSON Evaluators: Handling exact match comparison and JSON “fuzzy” matching via both deterministic and LLM-based methods.
– String and Similarity Evaluators: Modules that compute metrics such as Levenshtein distance and embedding similarity.
– Code Evaluation Modules: Separated by language – Python code evaluators using tools like Pyright, Mypy, and sandboxed execution (via E2B), and TypeScript evaluators that use the built-in type checker as well as E2B sandbox counterparts.
• Utility and Extraction Helpers:
– Code extraction utilities (e.g. extracting markdown code blocks or using an LLM to extract code) embedded within the evaluators.
– Config files, test suites, and common utility functions for evaluator orchestration.
• Integration and Orchestration:
– LangSmith Integration: Wrappers for logging inputs/outputs, experiment details, and evaluator results.
– Asynchronous Support: Both Python async wrappers (noted by functions/methods such as create_async_llm_as_judge) and TypeScript’s async evaluators.
• External Dependencies & Services:
– LLM Providers (e.g. OpenAI, Anthropic) – these are used within evaluators.
– E2B Sandboxed Execution: External sandbox environments (set up in the “sandbox” folder) for safely running/typing code.
– External utilities such as LangChain models and embeddings.
– CI/CD integrations (example: .github/workflows).
File Structure
– “Python Module” representing the python/openevals directory: Its components include subfolders like code, prompts, json, string, and tests, along with integration files such as py.typed and pyproject.toml.
– “JavaScript/TypeScript Module” representing the js folder: This contains similar submodules for code logic, tests, prompts, and configuration (package.json, tsconfigs).
• Additionally, represent shared configuration (like GitHub workflows, .gitignore, LICENSE, README) and the “sandbox” folder as a separate module that interfaces with both language components.
• A “static” folder which holds assets (such as images used in documentation or for LangSmith reporting).
Relationships and Data Flow
• How evaluator factory functions (e.g. create_llm_as_judge, create_json_match_evaluator, create_embedding_similarity_evaluator) branch into specific evaluation strategies.
• Connections between evaluators and external services:
• Use connectors to denote the common logging/integration layer (LangSmith integration) that receives feedback from various evaluators.
Architectural Patterns and Design Principles
• Modularity and separation of concerns: each evaluator tries to encapsulate a specific evaluation metric. You might group similar evaluators in containers (for example, one container for “LLM Evaluators”, another for “Code Evaluators”).
• Consistent interfaces for evaluators (inputs, outputs, and reference outputs). A simple “Evaluator Interface” component can denote this standardized design.
• Asynchronous design support across the library as a design pattern for scalability.
Technologies and Frameworks
• In the JavaScript branch, include TypeScript, Yarn (as seen from .yarn and configuration files), and integration with NodeJS packages.
• In the Python branch, show Python’s package configuration (pyproject.toml) and integration with LangChain, OpenAI, pyright, and mypy.
• External API connections (e.g., to OpenAI and Anthropic) along with the LangSmith and E2B sandbox services.
• Clear labels for each component, e.g. “LLM-as-Judge Evaluator (Python)” and “LLM-as-Judge Evaluator (JS)”, “Code Extraction Utility”, “TypeScript Type-Checker Evaluator”, etc.