# sleeepeer/poisonedrag architecture

> Architecture diagram of the GitHub repository sleeepeer/poisonedrag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sleeepeer/poisonedrag
- Repository: https://github.com/sleeepeer/poisonedrag

## Overview

##### Project Type & Purpose
- This is a research codebase supporting experiments on attacking RAG systems as detailed in a USENIX Security paper.
- Its purpose is to orchestrate evaluations of various language models and retrieval methods when subjected to knowledge-based corruption attacks. The experiments utilize different model configurations (e.g., GPT3.5, GPT4, PaLM2, LLaMA, Vicuna) and a retrieval system based on Contriever (a Facebook Research project).

##### Major Components
a. Experiment Orchestrators and Entry Points:
- Files like run.py, main.py, and evaluate_beir.py serve as the main drivers. They coordinate experiments and reproduction of results.
b. Dataset & Preprocessing:
- prepare_dataset.py fetches and prepares data (with the expected beir format) for experiments.
- The results directory indicates outcomes from both “adv_targeted” and “beir” experiment runs.
c. Model Configuration and API Integration:
- The model_configs directory holds JSON files with API key settings and parameters for each supported language model. This serves as an abstraction layer for model integration.
d. Retrieval Module (Contriever):
- Located in src/contriever_src, this component encapsulates the retrieval functionalities (loading data, indexing, evaluation, finetuning data generation, etc.). It’s an essential element in the RAG framework.
e. Attack Modules:
- The src/attack.py script embodies the logic for performing the knowledge corruption (or poisoning) attacks.
- The attack module interacts with both the retrieval module and the language model modules.
f. Language Model Interfaces:
- Found in src/models (files like GPT.py, Llama.py, PaLM2.py, Vicuna.py, and Model.py), these modules abstract the interaction with the corresponding external APIs or frameworks.
g. Utility Modules:
- src/utils.py and similar helper modules (e.g., prompts.py) support the orchestration logic, formatting, and any additional pre-/post-processing.

##### Relationships & Data Flows
- The experiment script loads configuration (API keys and hyperparameters) from model_configs.
- It then ensures datasets are present (or triggers prepare_dataset.py if needed) which feed into the evaluation.
- The retrieval component (Contriever in src/contriever_src) indexes and retrieves data while the attack module (src/attack.py) crafts corrupted input queries.
- The language model interface then sends these queries (using API keys from model_configs) to external services (e.g., OpenAI, Google’s generative AI, HuggingFace for LLaMA 2).
- Results from both retrieval and model outputs are aggregated and saved under results/ for further analysis.
- Evaluation routines may compare “truth” versus “attacked” results using additional evaluation and scoring modules.

##### Architectural Patterns & Design Principles
- Modularity: The project is clearly separated into modules for model interactions, retrieval logic, attack logic, data preparation, and evaluation.
- Configurability: With external JSON configuration files, the system supports pluggable components (different LLMs and retrieval models).
- Experiment Reproducibility: The orchestration script (run.py) encapsulates hyperparameters and diverse experiment settings explicitly.

b. Branch out from this layer:
- One branch for Dataset & Preprocessing (prepare_dataset.py).
- Another branch for the Attack & Evaluation Flow that connects to the LLM Interfaces.
- The Retrieval Component (src/contriever_src).
- The Attack Module (src/attack.py).
d. That both these modules interact with the Language Model Interface (src/models), which in turn communicates with external LLM services (noting API key usage from model_configs).
e. That the output from the LLMs is processed and stored in the “Results” directory.
f. Utility modules like src/utils.py, src/prompts.py, etc., as support blocks feeding into both the experiment orchestration and the model interfacing modules.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Experiment Orchestration Layer
    subgraph "Experiment Orchestration Layer"
        A1("run.py (Entry Point)"):::orchestrator
        A2("main.py (Entry Point)"):::orchestrator
        A3("evaluate_beir.py (Entry Point)"):::orchestrator
        A4("Utility Modules (src/utils.py & src/prompts.py)"):::utility
    end

    %% Configuration and Dataset Preparation
    subgraph "Configuration & Data Prep"
        B1("Model Configurations (model_configs)"):::config
        B2("Dataset Preparer (prepare_dataset.py)"):::data
    end

    %% Attack & Evaluation Flow
    subgraph "Attack & Evaluation Flow"
        C1("Retrieval Engine (src/contriever_src)"):::retrieval
        C2("Attack Module (src/attack.py)"):::attack
        C3("LLM Interfaces (src/models)"):::llm
    end

    %% External Services and Results
    subgraph "External & Results"
        D1("External LLM Services"):::external
        D2("Results Storage (results/...)"):::results
    end

    %% Relationships
    %% Experiment orchestration initiates configuration load and dataset preparation
    A1 -->|"loads"| B1
    A2 -->|"loads"| B1
    A1 -->|"triggers"| B2
    A2 -->|"triggers"| B2

    %% Utility modules support experiment orchestration and LLM interface
    A1 -->|"uses"| A4
    A2 -->|"uses"| A4
    C3 -->|"uses"| A4

    %% Data Flow from Dataset Prep to Attack & Evaluation Flow modules
    B2 -->|"provides data"| C1
    B2 -->|"provides data"| C2

    %% Retrieval and Attack modules both interact with LLM Interfaces
    C1 -->|"sends queries"| C3
    C2 -->|"sends corrupted queries"| C3

    %% LLM Interfaces call external LLM services
    C3 -->|"calls"| D1

    %% Aggregation: LLM responses routed to Results Storage
    D1 -->|"responses processed"| D2
    C3 -->|"outputs to"| D2

    %% Evaluation flow from evaluate_beir.py
    A3 -->|"evaluates outcomes"| D2

    %% Click Events
    click A1 "https://github.com/sleeepeer/poisonedrag/blob/main/run.py"
    click A2 "https://github.com/sleeepeer/poisonedrag/blob/main/main.py"
    click A3 "https://github.com/sleeepeer/poisonedrag/blob/main/evaluate_beir.py"
    click B1 "https://github.com/sleeepeer/poisonedrag/tree/main/model_configs"
    click B2 "https://github.com/sleeepeer/poisonedrag/blob/main/prepare_dataset.py"
    click C1 "https://github.com/sleeepeer/poisonedrag/tree/main/src/contriever_src"
    click C2 "https://github.com/sleeepeer/poisonedrag/blob/main/src/attack.py"
    click C3 "https://github.com/sleeepeer/poisonedrag/tree/main/src/models"
    click A4 "https://github.com/sleeepeer/poisonedrag/blob/main/src/utils.py"
    click A4 "https://github.com/sleeepeer/poisonedrag/blob/main/src/prompts.py"
    click D2 "https://github.com/sleeepeer/poisonedrag/tree/main/results"

    %% Styles
    classDef orchestrator fill:#f9f,stroke:#333,stroke-width:2px;
    classDef config fill:#ccf,stroke:#333,stroke-width:2px;
    classDef data fill:#cfc,stroke:#333,stroke-width:2px;
    classDef retrieval fill:#fc9,stroke:#333,stroke-width:2px;
    classDef attack fill:#f99,stroke:#333,stroke-width:2px;
    classDef llm fill:#9fc,stroke:#333,stroke-width:2px;
    classDef external fill:#ffc,stroke:#333,stroke-width:2px;
    classDef results fill:#cff,stroke:#333,stroke-width:2px;
    classDef utility fill:#ddd,stroke:#333,stroke-width:2px;
```
