How apartresearch/darkbench works
Project type and purpose
• DarkBench is an open-source Python benchmarking tool and dataset for detecting “dark patterns” in Large Language Model interactions.
• It ships as a library + command-line interface (CLI) with a built-in dataset (darkbench.jsonl) and pluggable scoring/registry modules.
File structure
• Top-level files:
• pyproject.toml & uv.lock – Python packaging & lockfile
• darkbench/ – core Python package
• reproduce/ – complete reproducible copy for artifact evaluation
• darkbench/ contains:
• _registry.py – plugin/registry of scoring functions
• dark_patterns.py – definitions of pattern categories
• scorer.py – logic to invoke LLMs & compute metrics
• task.py – orchestration of loading prompts & applying scorers
• darkbench.jsonl – dataset of 660 prompts
Extract insights from README
• Usage revolves around a CLI entrypoint: inspect eval darkbench --model provider/model
• External dependencies: model provider SDKs (openai, anthropic, etc.), environment variable API keys.
Main components (to represent
a. CLI Entrypoint
• “inspect eval darkbench” console script
b. Core Library
• Registry Module (_registry.py)
• Pattern Definitions (dark_patterns.py)
• Task Loader & Orchestrator (task.py)
• Scorer Engine (scorer.py)
c. Dataset
• darkbench.jsonl file
d. External Services
• LLM Provider APIs (OpenAI, Anthropic, Meta, etc.)
e. Output & Reporting
• Aggregated metrics, JSON/console outputs
Relationships & data flow
• The CLI invokes task.py
→ task.py reads darkbench.jsonl
→ For each prompt:
• Lookup scoring function via _registry
• Call scorer.py to send prompt to external API
• Receive LLM response
• Compute pattern-specific score
→ Collect per-task and per-category metrics
→ Emit results to console or file
Architectural patterns & design principles
• Plugin/Registry pattern: new scorers or pattern checks can be registered dynamically.
• Separation of concerns: dataset loading, scoring, pattern definitions, and CLI interface are modular.
• Extensibility: adding new model providers or patterns only requires implementing interfaces in scorer.py or dark_patterns.py and registering them.
Relevant technologies & libraries
• Python 3, pyproject.toml packaging (PEP 517/518)
• JSONL dataset file format
• External model SDKs (openai, anthropic, huggingface-transformers)
• CLI tool (“inspect” subcommand from some entrypoint)