# prompt-security/ps-fuzz architecture

> Architecture diagram of the GitHub repository prompt-security/ps-fuzz, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/prompt-security/ps-fuzz
- Repository: https://github.com/prompt-security/ps-fuzz

## Overview

##### Project Type and Purpose
- This is an open-source, Python-based command-line tool and library (“Prompt Fuzzer”) designed to dynamically test and harden system prompts for GenAI applications against a variety of simulated attacks.
- It exposes both a CLI interface and an interactive “playground” mode, uses multiple LLM providers via a pluggable chat-client abstraction, and supports extensibility through a registry of “attack” modules.

##### File Structure
Top-level artifacts and directories:
- .github/workflows – CI/CD and security scans (bandit, tests, release)
- build_package.sh, setup.py, MANIFEST.in – packaging and versioning
- ps_fuzz/ – core Python package
– __main__.py, cli.py, interactive_mode.py – entry points
– app_config.py, client_config.py, attack_config.py – config loaders
– chat_clients.py, langchain_integration.py – LLM provider adapters
– attack_loader.py, attack_registry.py, ps_logging.py – plugin/registry pattern and logging
– prompt_injection_fuzzer.py, work_progress_pool.py – orchestration and multi-threading
– attacks/ – individual attack implementations (each inherits a common interface)
– attack_data/ – CSV/Parquet files used by specific attacks
– results_table.py – result formatting & display
– util.py – shared utilities
- resources/, system_prompt.examples/ – static assets for demos
- tests/ – pytest unit tests

##### Insights from the README
- The system dynamically discovers attacks, dynamically selects LLM providers (OpenAI, Anthropic, etc.), and runs N attempts per attack in parallel threads.
- Output is a table of “Resilient” vs. “Broken” vs. “Errors.”
- Extensibility is a key design goal: new attacks and providers are added via registry modules.

##### Main Components and Their Interactions
a. CLI & Interactive Front End
- Parses command-line options (cli.py)
- In interactive mode (interactive_mode.py), launches a REPL-style prompt for iterative hardening
b. Configuration Layer
- app_config.py & attack_config.py read .env or env-vars
- client_config.py selects and configures the target LLM provider
c. Attack Registry & Loader
- attack_registry.py holds metadata for each attack (name, parameters)
- attack_loader.py dynamically imports attack implementations from ps_fuzz/attacks/
d. Core Fuzzer Engine (prompt_injection_fuzzer.py)
- Loads the system prompt and benchmark data
- For each registered attack:
– Generates attack prompts (using attack logic)
– Dispatches to the LLM via chat_clients.py adapter
– Collects responses and evaluates success/failure
e. Thread Pool & Orchestration (work_progress_pool.py)
- Manages a pool of worker threads, each running independent attack attempts
f. Result Aggregator & Display
- results_table.py formats results in a terminal table or CSV
g. LLM Provider Adapters (chat_clients.py & langchain_integration.py)
- Abstract base class defines send_prompt()
- Concrete subclasses implement OpenAI, Anthropic, etc.
h. Logging & Reporting (ps_logging.py)
- Centralized logging for debug levels, metrics, and error handling

##### Architectural Patterns and Design Principles
- Plugin/Registry Pattern for attacks and providers (loose coupling, discoverability)
- Strategy Pattern for chat client abstraction
- Separation of concerns: CLI vs. core engine vs. adapters vs. data
- Multi-threaded execution for parallel testing

##### Elements
- CLI / Interactive Mode
- Config Layer
- Attack Registry & Loader
- Fuzzer Engine
- Thread Pool / Workers
- Chat Client Adapters
- External LLM Services
- Result Aggregator / Display
- CLI → Engine
- Engine ↔ Config Layer
- Engine → Registry → Attack Modules
- Engine → Thread Pool → Workers → Chat Clients → External LLM APIs
- Engine → Result Aggregator → Display
- Blue for core components (Engine, Registry, Thread Pool)
- Green for extensibility points (Attack Modules, Chat Clients)
- Orange for external dependencies (LLM Providers, file-system data)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI & Interactive Front End
    subgraph "CLI & Interactive Front End"
        MAIN["__main__.py"]:::io
        CLI_P["cli.py"]:::io
        INT["interactive_mode.py"]:::io
        CHATUTIL["interactive_chat.py"]:::io
    end

    %% Configuration Layer
    subgraph "Configuration Layer"
        APP_CFG["app_config.py"]:::config
        ATTACK_CFG["attack_config.py"]:::config
        CLIENT_CFG["client_config.py"]:::config
        BUILD_SCRIPTS["build_package.sh / setup.py / MANIFEST.in"]:::config
    end

    %% Attack Registry & Loader
    subgraph "Attack Registry & Loader"
        REGISTRY["attack_registry.py"]:::core
        LOADER["attack_loader.py"]:::core
        subgraph "Attack Implementations" 
            A1["aim.py"]:::ext
            A2["base64_injection.py"]:::ext
            A3["complimentary_transition.py"]:::ext
            A4["custom_benchmark.py"]:::ext
            A5["dan.py"]:::ext
            A6["dynamic_test.py"]:::ext
            A7["ethical_compliance.py"]:::ext
            A8["harmful_behavior.py"]:::ext
            A9["self_refine.py"]:::ext
            A10["translation.py"]:::ext
            A11["typoglycemia.py"]:::ext
            A12["ucar.py"]:::ext
            A13["utils.py"]:::ext
        end
    end

    %% Core Fuzzer Engine
    FUZZER["prompt_injection_fuzzer.py"]:::core

    %% Thread Pool & Orchestration
    POOL["work_progress_pool.py"]:::core

    %% LLM Provider Adapters
    subgraph "LLM Provider Adapters"
        CLIENTS["chat_clients.py"]:::ext
        LANGCHAIN["langchain_integration.py"]:::ext
    end

    %% Result Aggregator & Display
    RESULTS["results_table.py"]:::core

    %% Logging & Reporting
    LOG["ps_logging.py"]:::core

    %% Shared Utilities & Assets
    UTIL["util.py"]:::core
    DATA["attack_data/"]:::external
    RESOURCES["resources/"]:::external
    SYS_PROMPTS["system_prompt.examples/"]:::external

    %% Testing Suite
    subgraph "Testing Suite"
        TEST1["tests/test_chat_clients.py"]:::test
        TEST2["tests/test_is_response_list.py"]:::test
        PYTEST["pytest.ini"]:::test
    end

    %% External LLM Services
    subgraph "External LLM Providers"
        OPENAI["OpenAI"]:::external
        ANTHROPIC["Anthropic"]:::external
        OTHER_LLM["Other LLMs"]:::external
    end

    %% Connections
    MAIN -->|loads| CLI_P
    CLI_P -->|"invokes"| FUZZER
    INT -->|"invokes"| FUZZER
    CHATUTIL -->|used by| INT

    APP_CFG -->|provides config| FUZZER
    ATTACK_CFG -->|provides config| REGISTRY
    CLIENT_CFG -->|provides config| CLIENTS
    BUILD_SCRIPTS -->|version & packaging| APP_CFG

    FUZZER -->|"loads attacks"| REGISTRY
    REGISTRY -->|"discovers modules"| LOADER
    LOADER -->|imports| A1 & A2 & A3 & A4 & A5 & A6 & A7 & A8 & A9 & A10 & A11 & A12 & A13
    FUZZER -->|"orchestrates"| POOL
    POOL -->|spawns| WORKER1["Worker"]:::core
    WORKER2["Worker"]:::core
    WORKER1 -->|calls| CLIENTS
    WORKER2 -->|calls| CLIENTS
    CLIENTS -->|sends prompt| OPENAI
    CLIENTS -->|sends prompt| ANTHROPIC
    CLIENTS -->|sends prompt| OTHER_LLM
    LANGCHAIN -->|alternate adapter| CLIENTS

    FUZZER -->|aggregates| RESULTS
    RESULTS -->|displays| CLI_P

    FUZZER -->|logs to| LOG
    UTIL -->|helpers| FUZZER

    DATA -->|input data| FUZZER
    SYS_PROMPTS -->|input prompts| FUZZER
    RESOURCES -->|demo assets| CLI_P

    TEST1 & TEST2 & PYTEST -->|test| CLIENTS & UTIL & REGISTRY & FUZZER

    %% Click Events
    click MAIN "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/__main__.py"
    click CLI_P "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/cli.py"
    click INT "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/interactive_mode.py"
    click CHATUTIL "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/interactive_chat.py"
    click APP_CFG "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/app_config.py"
    click ATTACK_CFG "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attack_config.py"
    click CLIENT_CFG "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/client_config.py"
    click BUILD_SCRIPTS "https://github.com/prompt-security/ps-fuzz/blob/main/build_package.sh"
    click REGISTRY "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attack_registry.py"
    click LOADER "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attack_loader.py"
    click A1 "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attacks/aim.py"
    click A2 "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attacks/base64_injection.py"
    click A3 "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attacks/complimentary_transition.py"
    click A4 "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attacks/custom_benchmark.py"
    click A5 "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attacks/dan.py"
    click A6 "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attacks/dynamic_test.py"
    click A7 "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attacks/ethical_compliance.py"
    click A8 "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attacks/harmful_behavior.py"
    click A9 "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attacks/self_refine.py"
    click A10 "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attacks/translation.py"
    click A11 "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attacks/typoglycemia.py"
    click A12 "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attacks/ucar.py"
    click A13 "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/attacks/utils.py"
    click FUZZER "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/prompt_injection_fuzzer.py"
    click POOL "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/work_progress_pool.py"
    click CLIENTS "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/chat_clients.py"
    click LANGCHAIN "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/langchain_integration.py"
    click RESULTS "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/results_table.py"
    click LOG "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/ps_logging.py"
    click UTIL "https://github.com/prompt-security/ps-fuzz/blob/main/ps_fuzz/util.py"
    click DATA "https://github.com/prompt-security/ps-fuzz/tree/main/ps_fuzz/attack_data/"
    click RESOURCES "https://github.com/prompt-security/ps-fuzz/tree/main/resources/"
    click SYS_PROMPTS "https://github.com/prompt-security/ps-fuzz/tree/main/system_prompt.examples/"
    click TEST1 "https://github.com/prompt-security/ps-fuzz/blob/main/tests/test_chat_clients.py"
    click TEST2 "https://github.com/prompt-security/ps-fuzz/blob/main/tests/test_is_response_list.py"
    click PYTEST "https://github.com/prompt-security/ps-fuzz/blob/main/pytest.ini"

    %% Styles
    classDef core fill:#cfe2f3,stroke:#6c8ebf
    classDef ext fill:#d9ead3,stroke:#6aa84f
    classDef external fill:#fce5cd,stroke:#e69138
    classDef config fill:#fff2cc,stroke:#b45f06
    classDef io fill:#ead1dc,stroke:#a64d79
    classDef test fill:#d0e0e3,stroke:#76a5af
```
