# huggingface/yourbench architecture

> Architecture diagram of the GitHub repository huggingface/yourbench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/huggingface/yourbench
- Repository: https://github.com/huggingface/yourbench

## Overview

##### Project type and purpose
- This is a Python-based, full-stack CLI library and framework called YourBench.
- Purpose: dynamically generate QA pairs, fine-tuning data, and evaluation benchmarks from arbitrary source documents via a configurable pipeline of parsing, chunking, LLM-driven stages, and dataset output.

##### File structure
- Top-level:
- yourbench/              ← core package
- example/                ← sample configs, inputs, prompts, and run scripts
- docs/                   ← documentation, whitepapers, design principles
- tests/                  ← unit and integration tests
- Makefile, pyproject.toml, CI workflows
- yourbench/ subdirectories:
- pipeline/               ← modular stages: ingestion, chunking, summarization, question_generation, rewriting, filtering
- utils/                  ← helpers: chunking_utils, config engine, dataset engine, inference interface, parsing engine
- prompts/                ← templated prompts for each stage, organized by functionality
- config_builder.py      ← transforms YAML into pipeline config
- app.py, main.py, __main__.py ← CLI entrypoint and orchestration
- example/ directory: multiple sub-folders, each with config.yaml, data, custom_prompts — demonstrates extensibility.

##### README for additional insights
- Describes core workflow: ingest documents → parse → chunk → generate summaries/questions → assemble HF dataset
- Emphasis on full configurability (toggle stages, swap models)
- External integration points: OpenAI, HF inference API, HF Hub upload
- Provides sample CLI command: uvx yourbench <dir> --model <model>

##### Diagramming the architecture
a. Main components
1. CLI Frontend
- entrypoint (yourbench CLI, uvx wrapper)
- config loader
2. Configuration Engine
- config_builder.py
- YAML config file
3. Pipeline Orchestrator
- handler.py (dispatch stages per config)
4. Stages (modules in pipeline/)
- Ingestion: read PDFs/text → intermediate markdown
- Chunking: break into tokens‐based or semantic chunks
- Summarization (optional)
- Question Generation
- Question Rewriting
- Citation Score Filtering
- Prepare for LightEval
5. Inference Layer
- inference_builders.py, inference_core.py, inference_tracking.py
- abstracts external LLM calls (OpenAI, HF API, local VLLM)
6. Prompt Templates
- organized under prompts/ by stage
7. Dataset Engine
- dataset_engine.py → builds a Hugging Face dataset, optional Hub upload
8. Utilities
- parsing_engine, url_utils, question_models, yourbench_card_template
9. Examples & Demo Apps
- example/ and Hugging Face Spaces

b. Relationships and interactions
- CLI → Configuration Engine → Pipeline Orchestrator
- Orchestrator sequentially invokes each Stage module
- Each Stage calls the Inference Layer, passing the appropriate prompt template
- Inference Layer returns LLM outputs → Passed back to stage → result passed to next stage
- Final output aggregated by Dataset Engine → local disk / HF Hub
- Utils modules provide shared services (e.g. parsing, chunk indexing, tracking)

c. Architectural patterns and design principles
- Modular pipeline: each stage is a plug-and-play component
- Separation of concerns: orchestration vs. stage logic vs. inference abstraction
- Configuration-driven execution: YAML config toggles stages, models, chunk size
- Extensibility: examples folder shows how to add new prompts, new data sources
- Test coverage: unit and integration tests target each component

d. Key technologies and libraries
- Python 3.12+, Click or argparse for CLI
- Hugging Face Datasets library for dataset output
- HTTP clients for OpenAI / HF Inference APIs
- uv (Astral) for zero-install CLI distribution
- PyTest for testing

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI Frontend
    subgraph "CLI Frontend"
        ME["__main__.py"]:::orchestration
        AP["app.py"]:::orchestration
        MA["main.py"]:::orchestration
    end

    %% Configuration Engine
    subgraph "Configuration Engine"
        CB["config_builder.py"]:::config
        CE["configuration_engine.py"]:::config
    end

    %% Pipeline Orchestrator
    OR["handler.py"]:::orchestration

    %% Pipeline Stages
    subgraph "Pipeline Stages"
        IG["ingestion.py"]:::pipeline
        CH["chunking.py"]:::pipeline
        SU["summarization.py"]:::pipeline
        QG["question_generation.py"]:::pipeline
        QR["question_rewriting.py"]:::pipeline
        CF["citation_score_filtering.py"]:::pipeline
        PL["prepare_lighteval.py"]:::pipeline
    end

    %% Inference Layer
    subgraph "Inference Layer"
        IB["inference_builders.py"]:::inference
        IC["inference_core.py"]:::inference
        IT["inference_tracking.py"]:::inference
    end

    %% Prompt Templates
    subgraph "Prompt Templates"
        IP["pdf_llm_prompt.md"]:::prompts
        SP1["summarization_user_prompt.md"]:::prompts
        SP2["combine_summaries_user_prompt.md"]:::prompts
        QGP["question_generation/*.md"]:::prompts
        QRP["question_rewriting/*.md"]:::prompts
        QA["qa/*.md"]:::prompts
    end

    %% Dataset Engine
    DE["dataset_engine.py"]:::dataset

    %% Examples & Demos
    EX["example/"]:::external

    %% Outputs
    LF["Local Filesystem"]:::external
    HF["Hugging Face Hub"]:::external

    %% Connections
    ME -->|runs CLI| CB
    ME -->|runs CLI| CE
    AP --> CB
    MA --> CE
    CB -->|YAML config| OR
    CE --> OR
    OR --> IG
    IG --> CH
    CH --> SU
    SU --> QG
    QG --> QR
    QR --> CF
    CF --> PL
    %% Inference interactions
    IG -->|LLM API calls| IB
    CH -->|LLM API calls| IB
    SU -->|LLM API calls| IB
    QG -->|LLM API calls| IB
    QR -->|LLM API calls| IB
    CF -->|LLM API calls| IB
    PL -->|LLM API calls| IB
    IB --> IC
    IC --> IT
    IT -->|returns output| IG
    %% Prompts usage
    IG --> IP
    SU --> SP1
    SU --> SP2
    QG --> QGP
    QR --> QRP
    QG --> QA
    %% Dataset
    PL --> DE
    DE --> LF
    DE --> HF
    %% Examples
    EX -->|sample configs| CB
    EX -->|custom prompts| IP
    EX -->|data| IG

    %% Click Events
    click ME "https://github.com/huggingface/yourbench/blob/main/yourbench/__main__.py"
    click AP "https://github.com/huggingface/yourbench/blob/main/yourbench/app.py"
    click MA "https://github.com/huggingface/yourbench/blob/main/yourbench/main.py"
    click CB "https://github.com/huggingface/yourbench/blob/main/yourbench/config_builder.py"
    click CE "https://github.com/huggingface/yourbench/blob/main/yourbench/utils/configuration_engine.py"
    click OR "https://github.com/huggingface/yourbench/blob/main/yourbench/pipeline/handler.py"
    click IG "https://github.com/huggingface/yourbench/blob/main/yourbench/pipeline/ingestion.py"
    click CH "https://github.com/huggingface/yourbench/blob/main/yourbench/pipeline/chunking.py"
    click SU "https://github.com/huggingface/yourbench/blob/main/yourbench/pipeline/summarization.py"
    click QG "https://github.com/huggingface/yourbench/blob/main/yourbench/pipeline/question_generation.py"
    click QR "https://github.com/huggingface/yourbench/blob/main/yourbench/pipeline/question_rewriting.py"
    click CF "https://github.com/huggingface/yourbench/blob/main/yourbench/pipeline/citation_score_filtering.py"
    click PL "https://github.com/huggingface/yourbench/blob/main/yourbench/pipeline/prepare_lighteval.py"
    click IB "https://github.com/huggingface/yourbench/blob/main/yourbench/utils/inference/inference_builders.py"
    click IC "https://github.com/huggingface/yourbench/blob/main/yourbench/utils/inference/inference_core.py"
    click IT "https://github.com/huggingface/yourbench/blob/main/yourbench/utils/inference/inference_tracking.py"
    click IP "https://github.com/huggingface/yourbench/blob/main/yourbench/prompts/ingestion/pdf_llm_prompt.md"
    click SP1 "https://github.com/huggingface/yourbench/blob/main/yourbench/prompts/summarization/summarization_user_prompt.md"
    click SP2 "https://github.com/huggingface/yourbench/blob/main/yourbench/prompts/summarization/combine_summaries_user_prompt.md"
    click QGP "https://github.com/huggingface/yourbench/tree/main/yourbench/prompts/question_generation/"
    click QRP "https://github.com/huggingface/yourbench/tree/main/yourbench/prompts/question_rewriting/"
    click QA "https://github.com/huggingface/yourbench/tree/main/yourbench/prompts/qa/"
    click DE "https://github.com/huggingface/yourbench/blob/main/yourbench/utils/dataset_engine.py"
    click EX "https://github.com/huggingface/yourbench/tree/main/example/"

    %% Styles
    classDef orchestration fill:#ccffcc,stroke:#333,stroke-width:1px
    classDef pipeline fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef inference fill:#ffdd99,stroke:#333,stroke-width:1px
    classDef config fill:#e5ccff,stroke:#333,stroke-width:1px
    classDef prompts fill:#f9ccff,stroke:#333,stroke-width:1px
    classDef dataset fill:#ffeecc,stroke:#333,stroke-width:1px
    classDef external fill:#dddddd,stroke:#333,stroke-width:1px
```
