# mr-dheer/detect-pretrain-code-original architecture

> Architecture diagram of the GitHub repository mr-dheer/detect-pretrain-code-original, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mr-dheer/detect-pretrain-code-original
- Repository: https://github.com/mr-dheer/detect-pretrain-code-original

## Overview

##### Project type and purpose
- This is a research-oriented, full-pipeline Python tool (CLI application) for evaluating “pretraining data detection” methods on large language models.
- It takes as input a benchmark dataset (WikiMIA or BookMIA), issues inference queries to “target” and “reference” LLMs (via OpenAI API or Hugging Face Transformers), computes the Min-K%-Prob detection score (and other baselines), then evaluates and stores results.

##### File structure
- Top-level scripts:
- run.py (orchestrates the end-to-end evaluation)
- process_data.py (loads and preprocesses dataset examples)
- src/ (core modules):
- options.py (parses CLI arguments: target_model, ref_model, data, length)
- eval.py (computes evaluation metrics and baselines)
- out/ (output directory: stores raw model logits/probs under subfolders for each model pairing and input folder structures)
- Configuration and metadata: README.md, LICENSE, .gitignore

##### Core components
a. User / CLI Layer
- run.py: entry point invoked with flags
- options.py: argument parsing
b. Data Layer
- process_data.py: loads dataset from Hugging Face (WikiMIA/BookMIA)
- Local cache or HF data store
c. Inference Layer
- Target Model Interface
- If OpenAI model → openai.api_key + OpenAI API endpoint
- If Hugging Face model → Transformers pipeline
- Reference Model Interface (always HF)
- Both interfaces return token‐level probabilities/logits
d. Detection Algorithm Layer
- Min‐K%‐Prob scorer (implements the paper’s algorithm)
- Additional baselines: PPL, Calibration, compression etc.
e. Evaluation Layer
- eval.py: computes metrics (accuracy, AUC) over predictions vs. membership labels
f. Storage / Output Layer
- src/out/<target_model>_<ref_model>/.../input/*.txt (raw scores/logits)
- Final CSV/JSON reports (could be added)

##### Component interactions and data flow
1. CLI user calls `python run.py --target_model … --ref_model … --data … --length …`
2. options.py → configuration object
3. process_data.py → fetch and preprocess examples
4. For each example:
- Send prompt to Target Model Interface
- Send prompt to Reference Model Interface
- Receive probabilities/logits
- Pass logits to Min-K%-Prob module & other baseline modules
- Write raw scores to src/out/...
5. Once all examples are scored → eval.py loads raw outputs, computes metrics → writes final evaluation

##### Architectural patterns and design principles
- **Pipeline pattern**: clear sequencing—data loading → inference → scoring → evaluation
- **Separation of concerns**:
- options.py for configuration
- process_data.py for data handling
- run.py for orchestration
- eval.py for metrics
- **Extensibility**: New models or baselines can plug into the inference or scoring stages

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        direction TB
        Run["run.py: Orchestration Script"]:::local
        Opt["src/options.py: CLI Arg Parser"]:::local
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DataSet["WikiMIA/BookMIA Dataset"]:::dataset
        Proc["process_data.py: DataLoader"]:::local
    end

    %% Inference Layer
    subgraph "Inference Layer"
        direction TB
        OpenAI["OpenAI API"]:::external
        HF["Hugging Face Transformers"]:::external
    end

    %% Detection Algorithm Layer
    subgraph "Detection Layer"
        direction TB
        MinK["Min-K%-Prob Scorer"]:::local
        Baselines["Other Baselines"]:::local
    end

    %% Storage Layer
    subgraph "Storage / Output Layer"
        direction TB
        OutDir["src/out/ : Raw Scores & Logs"]:::storage
    end

    %% Evaluation Layer
    subgraph "Evaluation Layer"
        direction TB
        Eval["src/eval.py: Metrics & Baselines Computation"]:::local
    end

    %% Connections
    Run -->|invokes| Opt
    Run -->|calls| Proc
    Proc -->|loads examples| DataSet
    Proc -->|outputs prompts| OpenAI
    Proc -->|outputs prompts| HF
    OpenAI -->|returns logits| MinK
    HF -->|returns logits| MinK
    OpenAI -->|returns logits| Baselines
    HF -->|returns logits| Baselines
    MinK -->|scores| OutDir
    Baselines -->|scores| OutDir
    OutDir -->|raw outputs| Eval
    Eval -->|final metrics| OutDir

    %% Click Events
    click Run "https://github.com/mr-dheer/detect-pretrain-code-original/blob/main/run.py"
    click Opt "https://github.com/mr-dheer/detect-pretrain-code-original/blob/main/src/options.py"
    click Proc "https://github.com/mr-dheer/detect-pretrain-code-original/blob/main/process_data.py"
    click Eval "https://github.com/mr-dheer/detect-pretrain-code-original/blob/main/src/eval.py"
    click OutDir "https://github.com/mr-dheer/detect-pretrain-code-original/tree/main/src/out/"

    %% Styles
    classDef local fill:#B3CDE0,stroke:#333,stroke-width:1px
    classDef dataset fill:#C7E9C0,stroke:#333,stroke-width:1px
    classDef external fill:#FEB24C,stroke:#333,stroke-width:1px
    classDef storage fill:#FEE391,stroke:#333,stroke-width:1px
```
