How tianyi-lab/superfiltering works
Project Type and Purpose
• This is an open‐source data‐processing toolkit (not a web or mobile app).
• Its goal is to filter large instruction‐tuning datasets using a small “weak” language model (e.g. GPT-2) and, optionally, a diversity module, then evaluate the filtered subset.
Main Architectural Layers
A. Data Layer
• Raw Instruction Data (Alpaca, Alpaca-GPT4 JSON files under data/)
• Annotated Data with IFD Scores (data/data_with_ifd/…)
• Filtered Subsets (outputs of selection and diversity steps)
• Evaluation Test Sets (evaluation/test_data/*.jsonl)
B. Processing Modules (Python Code under code_ifd/ and code_diversity_fla/)
1. IFD Scoring Module (data_analysis.py)
– Loads raw JSON, runs model inference via HuggingFace transformers to compute IFD scores.
2. Data Merger Module (put_analysis_to_data.py)
– Merges score tensors (from a PyTorch dump) back into JSON records.
3. Selection Module (select_data.py)
– Sorts and samples N% of data by IFD score.
4. Diversity Module (do_fla.py)
– Encodes the high-quality subset with sentence-transformers/all-MiniLM-L6-v2.
– Applies Facility Location Function via submodlib to pick a diverse 2%.
C. Orchestration Layer (Shell Scripts under scripts/ and evaluation/scripts/)
• step1_select_data_analysis_gpt2.sh → IFD scoring
• step2_put_analysis_to_data.sh → Merge scores
• step3_select_data.sh → Filter by score
• optional_select_data_plus_diversity.sh → Diversity step
• Evaluation scripts (do_eval_generation.sh, etc.) run generation & pairwise GPT4 evaluations
D. Evaluation Modules (evaluation/generation/*.py)
• Generation wrappers to produce model responses
• GPT4‐based pairwise comparison (eval.py, review_eval_score.py)
Component Relationships and Data Flow
• Raw Data → IFD Scoring Module → IFD Scores
• IFD Scores + Raw Data → Data Merger → Annotated Dataset
• Annotated Dataset → Selection Module → Filtered Dataset
• (Optional) Filtered Dataset → Diversity Module → Final Subset
• Final Subset → Model Fine-Tuning (external to this repo)
• Model Checkpoints + Test Sets → Evaluation Generation → GPT4 Evaluator → Evaluation Reports
Architectural Patterns
• Pipeline Pattern: Sequential data transforms (score → merge → select → diversify)
• Modular CLI: Each step isolated into scripts + small Python modules