# meg-tong/sycophancy-eval architecture

> Architecture diagram of the GitHub repository meg-tong/sycophancy-eval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/meg-tong/sycophancy-eval
- Repository: https://github.com/meg-tong/sycophancy-eval

## Overview

##### Project type and purpose
- This is an open-source evaluation toolkit for measuring “sycophantic” behavior in large language models (LLMs).
- It provides JSONL datasets of human–assistant dialogues, a utilities module to run model inference (sync/async), and an example notebook that ties it all together.

##### File structure
- Top level
- README.md – high-level description and usage instructions
- example.ipynb – Jupyter notebook illustrating a typical evaluation run
- utils.py – Python module with `inference` and `async_inference` functions plus any helper code (e.g., caching)
- datasets/
- answer.jsonl, are_you_sure.jsonl, feedback.jsonl – three separate evaluation datasets, each containing prompt/metadata/base keys

##### README for additional insights
- Describes the JSONL format (prompt, base, metadata)
- Recommends using asynchronous calls & caching via `await utils.async_inference`
- Mentions external LLM API endpoints (implied, e.g., OpenAI or Anthropic)

a. Dataset Store
- Three JSONL files under datasets/
b. Data Loader & Prompt Constructor
- Code in utils.py that reads JSONL, extracts `prompt` sequences, renders templates
- Rectangle labeled “PromptBuilder (utils.py)”
c. Inference Engine Interface
- `utils.inference` (sync) and `utils.async_inference` (async + caching)
- Rectangle labeled “LLMClient”
d. External LLM Service
e. Notebook / Evaluation Harness
- example.ipynb reading from PromptBuilder, invoking LLMClient, collecting responses, computing metrics
- Rectangle labeled “Evaluation Notebook”
f. (Optional) Caching Layer
- e.g., an in-memory or disk cache inside utils.py

##### Relationships & interactions
- Dataset Store → Data Loader
- Data Loader → Inference Engine Interface
- Inference Engine Interface ↔ External LLM Service
- Inference Engine Interface → Notebook
- Notebook → Reporting / Metrics

##### Important architectural patterns & design principles
- Separation of concerns: data vs. prompt construction vs. inference vs. analysis
- Extensibility: swap in new LLM APIs by modifying `utils.py`
- Async & caching: improve throughput and stability for large batch inference

##### Relevant technologies & frameworks
- Python 3.x
- JSONL for datasets
- Jupyter Notebook (example.ipynb)
- asyncio (for `async_inference`)
- External LLM APIs (OpenAI, Anthropic, etc.)

##### Detail level
- Three JSONL files as individual stacks or collectively under a “datasets” container

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Store
    DataStore["datasets/*.jsonl"]:::dataStore
    click DataStore "https://github.com/meg-tong/sycophancy-eval/tree/main/datasets/"

    %% Utils Module
    subgraph "Utils Module"
        PromptBuilder["PromptBuilder\n(utils.py)"]:::code
        SyncClient["SyncClient\n(inference)"]:::code
        AsyncClient["AsyncClient + Cache\n(async_inference)"]:::code
    end
    click PromptBuilder "https://github.com/meg-tong/sycophancy-eval/blob/main/utils.py"
    click SyncClient "https://github.com/meg-tong/sycophancy-eval/blob/main/utils.py"
    click AsyncClient "https://github.com/meg-tong/sycophancy-eval/blob/main/utils.py"

    %% External Service
    ExternalLLM["External LLM Service"]:::external

    %% Notebook
    Notebook["Evaluation Notebook\n(example.ipynb)"]:::notebook
    click Notebook "https://github.com/meg-tong/sycophancy-eval/blob/main/example.ipynb"

    %% Documentation
    Documentation["Project Documentation\n(README.md)"]:::code
    click Documentation "https://github.com/meg-tong/sycophancy-eval/blob/main/README.md"

    %% Metrics
    Metrics["Reporting / Metrics"]:::code

    %% Connections
    DataStore -->|"read JSONL prompts"| PromptBuilder
    PromptBuilder -->|"send prompt batches"| SyncClient
    PromptBuilder -->|"send prompt batches"| AsyncClient
    SyncClient -->|"API request/response"| ExternalLLM
    AsyncClient -->|"API request/response"| ExternalLLM
    SyncClient -->|"return generated responses + metadata"| Notebook
    AsyncClient -->|"return generated responses + metadata"| Notebook
    Notebook -->|"compute sycophancy scores, charts"| Metrics

    %% Styles
    classDef dataStore fill:#D6EAF8,stroke:#1B4F72,color:#1B4F72;
    classDef code fill:#D5F5E3,stroke:#196F3D,color:#196F3D;
    classDef external fill:#FDEBD0,stroke:#B9770E,color:#B9770E;
    classDef notebook fill:#EBDEF0,stroke:#6C3483,color:#6C3483;
```
