# tianyi-lab/superfiltering architecture

> Architecture diagram of the GitHub repository tianyi-lab/superfiltering, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tianyi-lab/superfiltering
- Repository: https://github.com/tianyi-lab/superfiltering

## Overview

##### Project Type and Purpose
- This is an open‐source data‐processing toolkit (not a web or mobile app).
- Its goal is to filter large instruction‐tuning datasets using a small “weak” language model (e.g. GPT-2) and, optionally, a diversity module, then evaluate the filtered subset.

##### Main Architectural Layers
A. Data Layer
- Raw Instruction Data (Alpaca, Alpaca-GPT4 JSON files under `data/`)
- Annotated Data with IFD Scores (`data/data_with_ifd/…`)
- Filtered Subsets (outputs of selection and diversity steps)
- Evaluation Test Sets (`evaluation/test_data/*.jsonl`)
B. Processing Modules (Python Code under `code_ifd/` and `code_diversity_fla/`)
1. IFD Scoring Module (`data_analysis.py`)
– Loads raw JSON, runs model inference via HuggingFace `transformers` to compute IFD scores.
2. Data Merger Module (`put_analysis_to_data.py`)
– Merges score tensors (from a PyTorch dump) back into JSON records.
3. Selection Module (`select_data.py`)
– Sorts and samples N% of data by IFD score.
4. Diversity Module (`do_fla.py`)
– Encodes the high-quality subset with `sentence-transformers/all-MiniLM-L6-v2`.
– Applies Facility Location Function via `submodlib` to pick a diverse 2%.
C. Orchestration Layer (Shell Scripts under `scripts/` and `evaluation/scripts/`)
- `step1_select_data_analysis_gpt2.sh` → IFD scoring
- `step2_put_analysis_to_data.sh` → Merge scores
- `step3_select_data.sh` → Filter by score
- `optional_select_data_plus_diversity.sh` → Diversity step
- Evaluation scripts (`do_eval_generation.sh`, etc.) run generation & pairwise GPT4 evaluations
D. Evaluation Modules (`evaluation/generation/*.py`)
- Generation wrappers to produce model responses
- GPT4‐based pairwise comparison (`eval.py`, `review_eval_score.py`)

##### Component Relationships and Data Flow
- Raw Data → IFD Scoring Module → IFD Scores
- IFD Scores + Raw Data → Data Merger → Annotated Dataset
- Annotated Dataset → Selection Module → Filtered Dataset
- (Optional) Filtered Dataset → Diversity Module → Final Subset
- Final Subset → Model Fine-Tuning (external to this repo)
- Model Checkpoints + Test Sets → Evaluation Generation → GPT4 Evaluator → Evaluation Reports

##### Architectural Patterns
- Pipeline Pattern: Sequential data transforms (score → merge → select → diversify)
- Modular CLI: Each step isolated into scripts + small Python modules

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        Raw["Raw Instruction Data"]:::data
        Annotated["Annotated Data with IFD Scores"]:::data
        Filtered["Filtered Dataset"]:::data
        Diverse["Diverse Subset"]:::data
        TestSets["Evaluation Test Sets"]:::data
    end

    %% Processing Modules
    subgraph "Processing Modules"
        IFD["IFD Scoring Module"]:::proc
        Merger["Data Merger Module"]:::proc
        Selector["Selection Module"]:::proc
        Diversity["Diversity Module"]:::proc
    end

    %% External Models & Libraries
    subgraph "External Models & Libraries"
        Transformers["Transformers (GPT-2)"]:::external
        Encoder["Sentence-Transformers"]:::external
        Submodlib["submodlib"]:::external
        GPT4["GPT-4 API"]:::external
    end

    %% Orchestration Scripts
    subgraph "Orchestration Scripts"
        Step1["step1_select_data_analysis_gpt2.sh"]:::script
        Step2["step2_put_analysis_to_data.sh"]:::script
        Step3["step3_select_data.sh"]:::script
        OptDiv["optional_select_data_plus_diversity.sh"]:::script
    end

    %% Evaluation Scripts & Modules
    subgraph "Evaluation Layer"
        EvalGen["do_eval_generation.sh"]:::script
        FullEval["do_eval.sh"]:::script
        ReviewScript["do_review_eval_score.sh"]:::script
        Wrap["eval_generation_wrap.py"]:::proc
        Driver["eva_generation.py"]:::proc
        EvalPair["eval.py"]:::proc
        ReviewEval["review_eval_score.py"]:::proc
    end

    %% Data Flows
    Raw -->|"JSON"| IFD
    IFD -->|"*.pt"| Merger
    Merger -->|"JSON"| Annotated
    Annotated -->|".json"| Selector
    Selector -->|"Filtered Data"| Filtered
    Filtered -->|"to Diversity"| Diversity
    Diversity -->|"Diverse Data"| Diverse

    %% Orchestration Flows
    Step1 --> IFD
    Step2 --> Merger
    Step3 --> Selector
    Step3 --> Filtered
    OptDiv --> Diversity

    %% External Dependencies
    IFD --> Transformers
    Diversity --> Encoder
    Diversity --> Submodlib

    %% Evaluation Flows
    Filtered --> EvalGen
    Diverse --> EvalGen
    TestSets --> Wrap
    EvalGen --> Wrap
    Wrap --> Driver
    Driver --> EvalPair
    EvalPair --> ReviewEval
    ReviewEval --> ReviewScript
    ReviewScript --> FullEval
    EvalPair --> GPT4

    %% Click Events
    click Raw "https://github.com/tianyi-lab/superfiltering/blob/main/data/alpaca_data.json"
    click Annotated "https://github.com/tianyi-lab/superfiltering/tree/main/data/data_with_ifd/"
    click TestSets "https://github.com/tianyi-lab/superfiltering/blob/main/evaluation/test_data/*.jsonl"
    click IFD "https://github.com/tianyi-lab/superfiltering/blob/main/code_ifd/data_analysis.py"
    click Merger "https://github.com/tianyi-lab/superfiltering/blob/main/code_ifd/put_analysis_to_data.py"
    click Selector "https://github.com/tianyi-lab/superfiltering/blob/main/code_ifd/select_data.py"
    click Diversity "https://github.com/tianyi-lab/superfiltering/blob/main/code_diversity_fla/do_fla.py"
    click Step1 "https://github.com/tianyi-lab/superfiltering/blob/main/scripts/step1_select_data_analysis_gpt2.sh"
    click Step2 "https://github.com/tianyi-lab/superfiltering/blob/main/scripts/step2_put_analysis_to_data.sh"
    click Step3 "https://github.com/tianyi-lab/superfiltering/blob/main/scripts/step3_select_data.sh"
    click OptDiv "https://github.com/tianyi-lab/superfiltering/blob/main/scripts/optional_select_data_plus_diversity.sh"
    click EvalGen "https://github.com/tianyi-lab/superfiltering/blob/main/evaluation/scripts/do_eval_generation.sh"
    click FullEval "https://github.com/tianyi-lab/superfiltering/blob/main/evaluation/scripts/do_eval.sh"
    click ReviewScript "https://github.com/tianyi-lab/superfiltering/blob/main/evaluation/scripts/do_review_eval_score.sh"
    click Wrap "https://github.com/tianyi-lab/superfiltering/blob/main/evaluation/generation/eval_generation_wrap.py"
    click Driver "https://github.com/tianyi-lab/superfiltering/blob/main/evaluation/generation/eva_generation.py"
    click EvalPair "https://github.com/tianyi-lab/superfiltering/blob/main/evaluation/generation/eval.py"
    click ReviewEval "https://github.com/tianyi-lab/superfiltering/blob/main/evaluation/generation/review_eval_score.py"

    %% Styles
    classDef data fill:#D0E6F8,stroke:#0366d6,stroke-width:1px;
    classDef proc fill:#D5F5E3,stroke:#28a745,stroke-width:1px;
    classDef external fill:#FCE5CD,stroke:#e36209,stroke-width:1px;
    classDef script fill:#E2E3E5,stroke:#6a737d,stroke-width:1px;
```
