# anthropics/evals architecture

> Architecture diagram of the GitHub repository anthropics/evals, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anthropics/evals
- Repository: https://github.com/anthropics/evals

## Overview

##### Project type and purpose
- This is an open‐source dataset repository for “Model-Written Evaluation Datasets.”
- It contains collections of JSONL files organized by topic (persona, sycophancy, advanced-ai-risk, winogender), plus prompts for few-shot generation.
- There is no application code—its primary purpose is to supply evaluation data for downstream model evaluation pipelines.

##### File structure
- Top-level:
- README.md, LICENSE, .gitignore
- Four topic folders: `persona/`, `sycophancy/`, `advanced-ai-risk/`, `winogenerated/`.
- `advanced-ai-risk/` further splits into:
- `human_generated_evals/` (ground-truth JSONL files)
- `lm_generated_evals/` (model-written JSONL files)
- `prompts_for_few_shot_generation/` (JSONL few-shot prompt templates)
- Other topics follow a simpler pattern: one folder per topic, each containing JSONL.
- No build scripts or code modules—pure data.

##### Key insights from README
- The data is intended for evaluating dialogue agents or other LMs.
- Consumers will clone the repo, select a dataset, feed its JSONL entries (prompts, questions) to an LM, collect responses, and compute metrics against reference answers or behavioral criteria.

##### Main components
a. Data Storage
– GitHub repository hosting the JSONL datasets (4 topic modules).
b. Evaluation Pipeline
– A data‐processing module (scripts or notebook) that:
1. Reads JSONL prompts & inputs
2. Optionally loads few-shot context templates
c. LLM Service(s)
– External APIs or local model servers (e.g., OpenAI, Anthropic, a fine-tuned dialogue agent)
d. Metrics & Analysis
– Comparison engine that computes quality, bias, or behavioral metrics
– Visualization/reporting component (notebooks, dashboards)
e. Reference Data
– Human‐written evaluations (in `advanced-ai-risk/human_generated_evals`) used to calibrate or compare model-written outputs

##### Relationships and interactions
- Pipeline → LLM Service (requests) → LLM Service → Pipeline (responses)
- Pipeline → Metrics & Analysis (result logging, comparison with human refs)
- Metrics & Analysis → Reporting (charts, tables)

##### Architectural patterns and principles
- Data-centric ETL pipeline: clear separation of “Extract” (read JSONL), “Transform” (format model input, wrap with few-shot context), “Load” (write responses)
- Modularity: each topic folder can be plugged in or swapped independently
- Extensibility: new topics or prompts can be added without touching pipeline code
- External dependency isolation: keep LLM calls abstracted behind an interface

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Storage Layer
    subgraph "Data Storage"
        direction TB
        DS_repo["GitHub Repo / JSONL Data"]:::static

        subgraph "Dataset Modules"
            direction TB
            AAR["advanced-ai-risk/"]:::static
            P["persona/"]:::static
            Syc["sycophancy/"]:::static
            Win["winogenerated/"]:::static
        end

        subgraph "Repository Metadata"
            direction TB
            README["README.md"]:::static
            LICENSE["LICENSE"]:::static
            GITIGNORE[".gitignore"]:::static
        end

        subgraph "advanced-ai-risk Subfolders"
            direction TB
            HGE["human_generated_evals/"]:::static
            LME["lm_generated_evals/"]:::static
            PFG["prompts_for_few_shot_generation/"]:::static
        end
    end

    %% Compute and External Services
    EPipeline["Evaluation Pipeline"]:::compute
    LLM["LLM Service<br>(External API)"]:::external
    Metrics["Metrics & Analysis"]:::compute
    Reporting["Reporting / Visualization"]:::compute

    %% Data Flow
    AAR -->|ingest JSONL| EPipeline
    P -->|ingest JSONL| EPipeline
    Syc -->|ingest JSONL| EPipeline
    Win -->|ingest JSONL| EPipeline
    README -->|repo info| EPipeline
    LICENSE -->|license info| EPipeline
    GITIGNORE -->|ignore rules| EPipeline

    EPipeline -->|send prompts| LLM
    LLM -->|return responses| EPipeline

    EPipeline -->|log responses| Metrics
    HGE -->|reference data| Metrics

    Metrics -->|generate charts| Reporting

    %% Click Events
    click AAR "https://github.com/anthropics/evals/tree/main/advanced-ai-risk/"
    click P "https://github.com/anthropics/evals/blob/main/persona/*.jsonl"
    click Syc "https://github.com/anthropics/evals/blob/main/sycophancy/*.jsonl"
    click Win "https://github.com/anthropics/evals/blob/main/winogenerated/*.jsonl"
    click HGE "https://github.com/anthropics/evals/tree/main/advanced-ai-risk/human_generated_evals/"
    click LME "https://github.com/anthropics/evals/tree/main/advanced-ai-risk/lm_generated_evals/"
    click PFG "https://github.com/anthropics/evals/tree/main/advanced-ai-risk/prompts_for_few_shot_generation/"
    click README "https://github.com/anthropics/evals/blob/main/README.md"
    click LICENSE "https://github.com/anthropics/evals/tree/main/LICENSE"
    click GITIGNORE "https://github.com/anthropics/evals/blob/main/.gitignore"

    %% Styles
    classDef static fill:#AED6F1,stroke:#21618C,color:#1B2631;
    classDef compute fill:#A9DFBF,stroke:#196F3D,color:#1B2631;
    classDef external fill:#F5B041,stroke:#B9770E,color:#1B2631;
```
