How mr-dheer/detect-pretrain-code-original works
Project type and purpose
• This is a research-oriented, full-pipeline Python tool (CLI application) for evaluating “pretraining data detection” methods on large language models.
• It takes as input a benchmark dataset (WikiMIA or BookMIA), issues inference queries to “target” and “reference” LLMs (via OpenAI API or Hugging Face Transformers), computes the Min-K%-Prob detection score (and other baselines), then evaluates and stores results.
File structure
• Top-level scripts:
• run.py (orchestrates the end-to-end evaluation)
• process_data.py (loads and preprocesses dataset examples)
• src/ (core modules):
• options.py (parses CLI arguments: target_model, ref_model, data, length)
• eval.py (computes evaluation metrics and baselines)
• out/ (output directory: stores raw model logits/probs under subfolders for each model pairing and input folder structures)
• Configuration and metadata: README.md, LICENSE, .gitignore
Core components
a. User / CLI Layer
• run.py: entry point invoked with flags
• options.py: argument parsing
b. Data Layer
• process_data.py: loads dataset from Hugging Face (WikiMIA/BookMIA)
• Local cache or HF data store
c. Inference Layer
• Target Model Interface
• If OpenAI model → openai.api_key + OpenAI API endpoint
• If Hugging Face model → Transformers pipeline
• Reference Model Interface (always HF)
• Both interfaces return token‐level probabilities/logits
d. Detection Algorithm Layer
• Min‐K%‐Prob scorer (implements the paper’s algorithm)
• Additional baselines: PPL, Calibration, compression etc.
e. Evaluation Layer
• eval.py: computes metrics (accuracy, AUC) over predictions vs. membership labels
f. Storage / Output Layer
• src/out/<target_model>_<ref_model>/.../input/*.txt (raw scores/logits)
• Final CSV/JSON reports (could be added)
Component interactions and data flow
1. CLI user calls python run.py --target_model … --ref_model … --data … --length …
2. options.py → configuration object
3. process_data.py → fetch and preprocess examples
4. For each example:
• Send prompt to Target Model Interface
• Send prompt to Reference Model Interface
• Receive probabilities/logits
• Pass logits to Min-K%-Prob module & other baseline modules
• Write raw scores to src/out/...
5. Once all examples are scored → eval.py loads raw outputs, computes metrics → writes final evaluation
Architectural patterns and design principles
• Pipeline pattern: clear sequencing—data loading → inference → scoring → evaluation
• Separation of concerns:
• options.py for configuration
• process_data.py for data handling
• run.py for orchestration
• eval.py for metrics
• Extensibility: New models or baselines can plug into the inference or scoring stages