How ai45lab/reef works
Project Type & Purpose
• Key use cases: model-fingerprinting (CKA), classifier training (linear/MLP/CNN/GCN), and comparative experiments (PCS, ICS).
File Structure
Top-level:
– LICENSE, README.md, requirements.txt (project metadata)
– src/ (all Python code, datasets, notebooks, helper scripts)
Under src/:
– datasets/ (.csv files for various benchmarks)
– scripts/ (shell wrappers: save_activation.sh, save_logits.sh)
– compute_cka.py, generate_activations.py, generate_head_activations.py, logits.py, pcs.py, ics.py, train_cls.py, train_cls_gcn.py, transfer_cls.py, model.py, utils.py
– plot.ipynb (visualization notebook)
Read README for Architecture Insights
– Data sources: CSV benchmarks + pre-trained LLM checkpoints (e.g., llama-2-7b, vicuna-7b)
– Modules invoked by shell scripts (activation & logits extraction)
– Main computational steps: generate activations → compute CKA → train/transfer classifiers → plot
– Comparative experiments: PCS, ICS
Main Components & Their Interactions
A. Data Layer
• Raw datasets (.csv)
• Pre-trained models (Hugging Face checkpoint downloads)
B. Extraction Layer
• generate_activations.py (invoked by save_activation.sh)
• generate_head_activations.py (for per-head features)
• logits.py (invoked by save_logits.sh)
C. Similarity Computation
• compute_cka.py → imports model.py + utils.py
• Outputs a CKA similarity matrix
D. Classifier Module
• train_cls.py (linear/MLP/CNN)
• train_cls_gcn.py (graph-based classifier)
• transfer_cls.py (applies pre-trained classifier to suspect models)
E. Comparative Experiments
• pcs.py, ics.py (other fingerprinting methods)
F. Visualization
• plot.ipynb (heatmaps, graphs)
G. Orchestration
• Shell scripts in src/scripts/
• CLI arguments in Python entrypoints
Architectural Patterns & Design Principles
– Modular Python package: each major function (activation, CKA, classification) lives in its own script
– Separation of concerns: data ingestion, feature extraction, similarity computation, classifier training, visualization
– Reusable utility code in utils.py and centralized model loading in model.py
– Shell scripts as lightweight pipelines for reproducibility