# imagination-research/lbt architecture

> Architecture diagram of the GitHub repository imagination-research/lbt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/imagination-research/lbt
- Repository: https://github.com/imagination-research/lbt

## Overview

##### Project Type and Purpose
- This is a research codebase and experimental toolkit (open-source tool) for exploring “Learning by Teaching” (LbT) in large language models (LLMs). Rather than a traditional full-stack app, it’s a Python library plus CLI scripts that orchestrate data preparation, LbT-based prompting experiments (M1), DPO fine-tuning (M2), and prompt optimization (M3). It integrates with external LLM API endpoints (Azure OpenAI, HuggingFace).

##### File Structure Analysis
- Top-level directories:
- lbt/ : core Python package implementing LbT abstractions (models, exam maker/prompter/scorer, dataset adapters, utilities).
- scripts/: CLI entry points for dataset preparation and experiments (math and code variants).
- examples/config/: YAML config templates for M1 experiments (math, code).
- examples/datasets/: placeholder for processed data.
- tests/: unit tests for core modules (e.g., math extraction, exam-maker).
- lbt/patch/: patch for LMOps prompt_optimization (M3).
- pyproject.toml: packaging/build.
- pre-commit, pylint config: linting/CI.

##### README Insights
- Describes three methods:
- M1: search-based generation pipeline that uses “teaching problems” → generate rationales → student exams → LbT scoring and selection.
- M2: use LbT scores to fine-tune the teacher model via Direct Preference Optimization.
- M3: prompt optimization pipeline based on a patched Microsoft LMOps codebase.
- Emphasizes components: exam_maker, exam_prompter, model interfaces, exam_scorer, plus scripts for each stage.

a. Main Components
- CLI Layer (scripts/exam.py, code_exam.py, prepare_datasets.py, search_rationale.py)
- Core Library (lbt package):
– Dataset Adapters (fetching, formatting LeetCode & math).
– ExamMaker, ExamPrompter, ExamScorer classes.
– Model Interfaces (AzureOpenAI, HuggingFace).
– Utilities (logging, registry).
- External Services: Azure OpenAI API, HuggingFace Transformer Inference, LeetCode website/API.
- Third-party Codebases: DPO codebase (M2), patched LMOps/prompt_optimization (M3).
- Config Store: YAML files under examples/config.
- Data Store: Local file system for datasets and experiment outputs.
b. Relationships and Interactions
1. User invokes a CLI script with a YAML config →
2. CLI loads config → uses Dataset Adapters to load teaching/exam datasets.
3. CLI instantiates ExamMaker/Prompter/Scorer and Model Interface(s).
4. Prompter builds prompts (TP + TR or EP) → Model Interface sends to external API (Azure/HF) → receives responses.
5. Scorer parses model output → computes LbT scores → writes outputs.
6. Search_rationale script reads outputs → selects best teaching answer (M1) or aggregates for submission.
7. M2 pipeline: outputs + scores feed into DPO training codebase.
8. M3 pipeline: user applies patch, then calls LMOps main.py with config.py → internal prompt optimization loop with teacher/student API calls.
c. Architectural Patterns & Design Principles
- Plugin/Registry: dynamic instantiation of model, prompter, scorer by type string.
- Config-Driven orchestration: behavior controlled entirely by YAML.
- Separation of Concerns: core LbT logic (lbt/) decoupled from CLI adapters (scripts/) and external code (DPO, LMOps).
- Modular Dataset Adapters: code and math share a common adapter interface.
d. Key Technologies
- Python 3
- Azure OpenAI REST API
- HuggingFace Transformers
- YAML for configuration
- Third-party repos: direct-preference-optimization, Microsoft LMOps prompt_optimization

##### Detailed Capture of Architectural Elements
- Configuration points (e.g., registry lookup by “model_type”).
- Parallel splits and batch generation in M1 (can show loops or annotations).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration Store
    subgraph "Configuration Store"
        CFG["YAML Config Templates"]:::config
    end

    %% CLI Layer
    subgraph "CLI Layer" 
        EXAM["scripts/exam.py"]:::cli
        CODE_EXAM["scripts/code_exam.py"]:::cli
        CODE_PREPARE["scripts/code/prepare_datasets.py"]:::cli
        CODE_TEACH["scripts/code/prepare_teaching_datasets.py"]:::cli
        CODE_SEARCH["scripts/code/search_rationale.py"]:::cli
        MATH_PREPARE["scripts/math/prepare_datasets.py"]:::cli
        MATH_TEACH["scripts/math/prepare_teaching_dataset.py"]:::cli
        MATH_SEARCH["scripts/math/search_rationale.py"]:::cli
    end

    %% Core Library
    subgraph "Core Library (lbt/)"
        subgraph "Dataset Adapters" 
            CODE_ADAPT["code_dataset.py"]:::lib
            MATH_ADAPT["math_dataset.py"]:::lib
            LEE_ADAPT["leetcode_sub/"]:::lib
            DS_UTILS["utils/"]:::lib
        end
        EXAM_MAKER["lbt/exam_maker.py"]:::lib
        EXAM_SCORER["lbt/exam_scorer.py"]:::lib
        MODEL_BASE["lbt/models/base.py"]:::lib
        REGISTRY["lbt/utils/registry.py"]:::lib
        LOG_UTIL["lbt/utils/log.py"]:::lib
        BASE_UTIL["lbt/base.py"]:::lib
        EXAM_PROMPTER["ExamPrompter"]:::lib
        MODEL_IFACE["Model Interface"]:::lib
    end

    %% External Services
    subgraph "External Services"
        AZURE["Azure OpenAI API"]:::external
        HF["HuggingFace Inference"]:::external
        LEETCODE["LeetCode API"]:::external
    end

    %% Data Store
    subgraph "Data Store"
        DATA_FS["Local File System"]:::data
    end

    %% Third-party Code
    subgraph "Third-party Code"
        DPO["DirectPreferenceOptimization Codebase"]:::third
        LMOPS["Microsoft LMOps Prompt Optimization"]:::third
        PATCH["lbt/patch/lmops-lbtm3.patch"]:::third
    end

    %% Flows
    CFG -->|"loads config"| EXAM
    CFG --> CODE_EXAM
    CFG --> CODE_PREPARE
    CFG --> CODE_TEACH
    CFG --> CODE_SEARCH
    CFG --> MATH_PREPARE
    CFG --> MATH_TEACH
    CFG --> MATH_SEARCH

    EXAM -->|"uses adapter"| CODE_ADAPT
    CODE_EXAM --> CODE_ADAPT
    CODE_PREPARE --> CODE_ADAPT
    CODE_TEACH --> CODE_ADAPT
    CODE_SEARCH --> EXAM_SCORER

    MATH_PREPARE --> MATH_ADAPT
    MATH_TEACH --> MATH_ADAPT
    MATH_SEARCH --> EXAM_SCORER

    CODE_ADAPT --> DATA_FS
    MATH_ADAPT --> DATA_FS

    EXAM --> EXAM_MAKER
    EXAM --> EXAM_PROMPTER
    EXAM --> EXAM_SCORER

    EXAM_PROMPTER -->|"sends prompt"| MODEL_IFACE
    MODEL_IFACE -->|"API call"| AZURE
    MODEL_IFACE -->|"API call"| HF
    CODE_ADAPT -->|"fetches data"| LEETCODE
    EXAM_SCORER -->|"writes scores"| DATA_FS

    EXAM_SCORER --> DPO
    EXAM_SCORER --> LMOPS
    PATCH --> LMOPS

    %% Click Events
    click EXAM "https://github.com/imagination-research/lbt/blob/master/scripts/exam.py"
    click CODE_EXAM "https://github.com/imagination-research/lbt/blob/master/scripts/code_exam.py"
    click CODE_PREPARE "https://github.com/imagination-research/lbt/blob/master/scripts/code/prepare_datasets.py"
    click CODE_TEACH "https://github.com/imagination-research/lbt/blob/master/scripts/code/prepare_teaching_datasets.py"
    click CODE_SEARCH "https://github.com/imagination-research/lbt/blob/master/scripts/code/search_rationale.py"
    click MATH_PREPARE "https://github.com/imagination-research/lbt/blob/master/scripts/math/prepare_datasets.py"
    click MATH_TEACH "https://github.com/imagination-research/lbt/blob/master/scripts/math/prepare_teaching_dataset.py"
    click MATH_SEARCH "https://github.com/imagination-research/lbt/blob/master/scripts/math/search_rationale.py"
    click CODE_ADAPT "https://github.com/imagination-research/lbt/blob/master/lbt/datasets_adapter/code_dataset.py"
    click MATH_ADAPT "https://github.com/imagination-research/lbt/blob/master/lbt/datasets_adapter/math_dataset.py"
    click LEE_ADAPT "https://github.com/imagination-research/lbt/tree/master/lbt/datasets_adapter/leetcode_sub/"
    click DS_UTILS "https://github.com/imagination-research/lbt/tree/master/lbt/datasets_adapter/utils/"
    click EXAM_MAKER "https://github.com/imagination-research/lbt/blob/master/lbt/exam_maker.py"
    click EXAM_SCORER "https://github.com/imagination-research/lbt/blob/master/lbt/exam_scorer.py"
    click MODEL_BASE "https://github.com/imagination-research/lbt/blob/master/lbt/models/base.py"
    click REGISTRY "https://github.com/imagination-research/lbt/blob/master/lbt/utils/registry.py"
    click LOG_UTIL "https://github.com/imagination-research/lbt/blob/master/lbt/utils/log.py"
    click BASE_UTIL "https://github.com/imagination-research/lbt/blob/master/lbt/base.py"
    click PATCH "https://github.com/imagination-research/lbt/blob/master/lbt/patch/lmops-lbtm3.patch"
    click CFG "https://github.com/imagination-research/lbt/tree/master/examples/config/"

    %% Styles
    classDef config fill:#f9f,stroke:#333
    classDef cli fill:#bbf,stroke:#333
    classDef lib fill:#bfb,stroke:#333
    classDef external fill:#fdb,stroke:#333
    classDef data fill:#ddd,stroke:#333
    classDef third fill:#f99,stroke:#333
```
