# apartresearch/darkbench architecture

> Architecture diagram of the GitHub repository apartresearch/darkbench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apartresearch/darkbench
- Repository: https://github.com/apartresearch/darkbench

## Overview

##### Project type and purpose
- DarkBench is an open-source Python benchmarking tool and dataset for detecting “dark patterns” in Large Language Model interactions.
- It ships as a library + command-line interface (CLI) with a built-in dataset (darkbench.jsonl) and pluggable scoring/registry modules.

##### File structure
- Top-level files:
- pyproject.toml & uv.lock – Python packaging & lockfile
- darkbench/ – core Python package
- reproduce/ – complete reproducible copy for artifact evaluation
- darkbench/ contains:
- _registry.py – plugin/registry of scoring functions
- dark_patterns.py – definitions of pattern categories
- scorer.py – logic to invoke LLMs & compute metrics
- task.py – orchestration of loading prompts & applying scorers
- darkbench.jsonl – dataset of 660 prompts

##### Extract insights from README
- Usage revolves around a CLI entrypoint: `inspect eval darkbench --model provider/model`
- External dependencies: model provider SDKs (openai, anthropic, etc.), environment variable API keys.

##### Main components (to represent
a. CLI Entrypoint
- “inspect eval darkbench” console script
b. Core Library
- Registry Module (_registry.py)
- Pattern Definitions (dark_patterns.py)
- Task Loader & Orchestrator (task.py)
- Scorer Engine (scorer.py)
c. Dataset
- darkbench.jsonl file
d. External Services
- LLM Provider APIs (OpenAI, Anthropic, Meta, etc.)
e. Output & Reporting
- Aggregated metrics, JSON/console outputs

##### Relationships & data flow
- The CLI invokes task.py
→ task.py reads darkbench.jsonl
→ For each prompt:
- Lookup scoring function via _registry
- Call scorer.py to send prompt to external API
- Receive LLM response
- Compute pattern-specific score
→ Collect per-task and per-category metrics
→ Emit results to console or file

##### Architectural patterns & design principles
- Plugin/Registry pattern: new scorers or pattern checks can be registered dynamically.
- Separation of concerns: dataset loading, scoring, pattern definitions, and CLI interface are modular.
- Extensibility: adding new model providers or patterns only requires implementing interfaces in scorer.py or dark_patterns.py and registering them.

##### Relevant technologies & libraries
- Python 3, pyproject.toml packaging (PEP 517/518)
- JSONL dataset file format
- External model SDKs (openai, anthropic, huggingface-transformers)
- CLI tool (“inspect” subcommand from some entrypoint)

## Diagram (Mermaid)

```mermaid
flowchart TB
    classDef core fill:#add8e6,stroke:#000;
    classDef data fill:#90ee90,stroke:#000;
    classDef external fill:#ffcc99,stroke:#000;
    classDef cli fill:#d3d3d3,stroke:#000;
    classDef legend fill:#ffffff,stroke:#000,stroke-dasharray: 2 2;

    CLI[/CLI Entrypoint/]:::cli
    Task["Task Loader & Orchestrator\ntask.py"]:::core
    Registry["Registry Module\n_registry.py"]:::core
    Patterns["Pattern Definitions\ndark_patterns.py"]:::core
    Scorer["Scorer Engine\nscorer.py"]:::core
    Dataset[("darkbench.jsonl")]:::data
    External(("LLM Provider APIs")):::external
    Output["Output & Reporting\n(JSON/Console)"]:::core

    CLI -->|"invoke"| Task
    Task -->|"load prompts"| Dataset
    Task -->|"lookup scorer"| Registry
    Registry -->|"get function"| Scorer
    Task -->|"use definitions"| Patterns
    Scorer -->|"send prompt"| External
    External -->|"LLM response"| Scorer
    Scorer -->|"score(prompt)"| Task
    Task -->|"emit results"| Output

    subgraph Legend
        A["Core Library"]:::core
        B["Dataset"]:::data
        C["External Services"]:::external
        D["CLI Entrypoint"]:::cli
    end
    class A,B,C,D legend

    click CLI "https://github.com/apartresearch/darkbench/blob/main/pyproject.toml"
    click Registry "https://github.com/apartresearch/darkbench/blob/main/darkbench/_registry.py"
    click Patterns "https://github.com/apartresearch/darkbench/blob/main/darkbench/dark_patterns.py"
    click Task "https://github.com/apartresearch/darkbench/blob/main/darkbench/task.py"
    click Scorer "https://github.com/apartresearch/darkbench/blob/main/darkbench/scorer.py"
    click Dataset "https://github.com/apartresearch/darkbench/blob/main/darkbench/darkbench.jsonl"
```
