# magpie-align/magpie architecture

> Architecture diagram of the GitHub repository magpie-align/magpie, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/magpie-align/magpie
- Repository: https://github.com/magpie-align/magpie

## Overview

##### Project Type and Purpose
- Magpie is an open-source machine-learning tool for large-scale alignment data synthesis.
- It orchestrates prompt-based generation, tagging, filtering, and packaging of instruction–response pairs by driving aligned LLMs (e.g., Llama-3, Qwen2) via shell scripts and Python code.

##### File Structure Analysis
- Top-level directories:
– configs: JSON model configuration
– scripts: shell scripts invoking different model variants for data generation and tagging
– data_sft & data_po: Jupyter notebooks and example JSONL for supervised-fine-tuning (SFT) and preference-optimization (PO) data pipelines
– exp: Python utilities for filtering, computing distances (FAISS), reward calculation, string utilities
– recipes: YAML recipes for fine-tuning on various models
- Supporting files:
– README.md: project overview, architecture summary, usage instructions
– requirements.txt: Python dependencies
– demo.ipynb: interactive example

##### Additional Architectural Insights from README
- Magpie’s core workflow is batched data generation → multi-turn expansion (optional) → tagging → concatenation/conversion → repetition removal → custom filtering → preference data generation → dataset export.
- External dependencies/services: Huggingface model hub (LLM weights), GPU clusters (A100/RTX), FAISS for similarity search, Axolotl for fine-tuning format support.

##### Components
a. Configuration Manager
– configs/model_configs.json
– recipes/*.yaml
b. Generation Engine
– scripts/magpie*.sh (single-turn, multi-turn)
– demo.ipynb
– External Service: 🤗 Huggingface API / vLLM
c. Tagging Service
– scripts/unitag.sh
– exp/str_utils.py
d. Data Processing Pipeline
– data_sft/data_concatenation.ipynb
– data_sft/data_filter.ipynb
– exp/gen_dis.py (FAISS indexing)
– data_po/process_po.ipynb
e. Storage & Export
– Intermediate JSONL/ShareGPT files
– Final datasets uploaded to Huggingface Datasets
f. Optional Extensions
– Preference-Optimization (PO) scripts & notebooks
– Custom filters

##### Relationships and Data Flows
- Config Manager feeds Generation Engine with model parameters & prompts
- Generation Engine calls LLMs → raw instruction-response JSONL
- Tagging Service annotates raw JSONL → adds quality, safety, reward labels
- Data Processing Pipeline:
– concatenates multiple JSONLs → ShareGPT format
– removes duplicates via FAISS → filtered JSONL
- Preference Data Pipeline: generates multi-response sets → computes reward scores → processes into PO JSONL
- Storage & Export: final JSONL uploaded to external storage (Huggingface)

##### Architectural Patterns & Design Principles
- Modular pipeline: each stage (generation, tagging, filtering, packaging) is decoupled
- Configuration-driven: adding new models or generation settings via recipes and JSON configs
- Extensible scripting layer: shell + Python blend for easy customization
- Notebook-friendly: Jupyter notebooks for exploration and ad hoc pipeline steps

##### Technologies and Frameworks
- Shell scripting (bash) for orchestration
- Python for data processing (FAISS, vLLM, Huggingface Transformers)
- Jupyter notebooks for interactive workflows
- JSON/YAML for config and data interchange
- Axolotl for fine-tuning data format

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Configuration Manager
    subgraph "Configuration Manager"
        direction TB
        ConfigJSON["model_configs.json"]:::config
        Recipes["YAML Recipes"]:::config
    end

    %% Generation Engine
    subgraph "Generation Engine"
        direction TB
        ShellScripts["magpie*.sh"]:::gen
        DemoNotebook["demo.ipynb"]:::notebook
        HFService["Huggingface API"]:::external
        GPUCluster["GPU Cluster"]:::external
    end

    %% Tagging Service
    subgraph "Tagging Service"
        direction TB
        UnitagScript["unitag.sh"]:::gen
        StrUtils["str_utils.py"]:::gen
    end

    %% Data Processing Pipeline
    subgraph "Data Processing Pipeline"
        direction TB
        ConcatNotebook["data_concatenation.ipynb"]:::notebook
        FilterNotebook["data_filter.ipynb"]:::notebook
        FAISSIndex["gen_dis.py"]:::processing
    end

    %% Preference-Optimization Pipeline
    subgraph "Preference-Optimization Pipeline"
        direction TB
        POProcess["process_po.ipynb"]:::notebook
        POMultiRes["gen_po_multi_res.py"]:::processing
        PORewards["gen_po_rewards.py"]:::processing
    end

    %% Supervised Fine-Tuning Pipeline
    subgraph "SFT Pipeline"
        direction TB
        ExampleJSONL["example_instructions.jsonl"]:::processing
        Example5Res["example_instructions_5res.json"]:::processing
        ExampleArmor["example_instructions_5res_armorm.json"]:::processing
    end

    %% Storage & Export
    subgraph "Storage & Export"
        direction TB
        Intermediate[(Intermediate JSONL/ShareGPT)]:::external
        Final[(Final Datasets)]:::external
        HFData["Huggingface Datasets"]:::external
    end

    %% Flows
    ConfigJSON -->|feeds parameters| ShellScripts
    Recipes -->|provides prompts| ShellScripts
    ShellScripts -->|"calls LLMs"| HFService
    HFService -->|"raw JSONL"| Intermediate
    GPUCluster ---> HFService

    Intermediate --> UnitagScript
    StrUtils --> UnitagScript
    UnitagScript -->|"annotated JSONL"| Intermediate

    Intermediate --> ConcatNotebook
    ConcatNotebook --> FAISSIndex
    FAISSIndex --> FilterNotebook
    FilterNotebook --> Final

    Final --> HFData

    %% Optional PO flow
    Intermediate --> POProcess
    POProcess --> POMultiRes
    POMultiRes --> PORewards
    PORewards --> Final

    %% Optional SFT artifacts
    ExampleJSONL --> Final
    Example5Res --> Final
    ExampleArmor --> Final

    %% Click Events
    click ConfigJSON "https://github.com/magpie-align/magpie/blob/main/configs/model_configs.json"
    click Recipes "https://github.com/magpie-align/magpie/tree/main/recipes/"
    click ShellScripts "https://github.com/magpie-align/magpie/tree/main/scripts/"
    click DemoNotebook "https://github.com/magpie-align/magpie/blob/main/demo.ipynb"
    click UnitagScript "https://github.com/magpie-align/magpie/blob/main/scripts/unitag.sh"
    click StrUtils "https://github.com/magpie-align/magpie/blob/main/exp/str_utils.py"
    click ConcatNotebook "https://github.com/magpie-align/magpie/blob/main/data_sft/data_concatenation.ipynb"
    click FilterNotebook "https://github.com/magpie-align/magpie/blob/main/data_sft/data_filter.ipynb"
    click FAISSIndex "https://github.com/magpie-align/magpie/blob/main/exp/gen_dis.py"
    click POProcess "https://github.com/magpie-align/magpie/blob/main/data_po/process_po.ipynb"
    click POMultiRes "https://github.com/magpie-align/magpie/blob/main/exp/gen_po_multi_res.py"
    click PORewards "https://github.com/magpie-align/magpie/blob/main/exp/gen_po_rewards.py"
    click ExampleJSONL "https://github.com/magpie-align/magpie/blob/main/data_po/example_instructions.jsonl"
    click Example5Res "https://github.com/magpie-align/magpie/blob/main/data_po/example_instructions_5res.json"
    click ExampleArmor "https://github.com/magpie-align/magpie/blob/main/data_po/example_instructions_5res_armorm.json"
    click HFData "https://github.com/magpie-align/magpie/tree/main/Huggingface Datasets"

    %% Styles
    classDef config fill:#E0F7FA,stroke:#0288D1,color:#000
    classDef gen fill:#E8F5E9,stroke:#2E7D32,color:#000
    classDef processing fill:#FFF3E0,stroke:#EF6C00,color:#000
    classDef notebook fill:#F3E5F5,stroke:#6A1B9A,color:#000
    classDef external fill:#EEEEEE,stroke:#757575,color:#000
```
