How utkusen/promptmap works
Main Components
• CLI Interface & Argument Parser: Represents the command-line entry point (promptmap2.py) where users specify parameters like model type, rules, iterations, firewall mode, output file, etc.
• Rule Loader & Parser: This module reads the test rules from the rules/ directory (a number of YAML files) and converts them into internal representations.
• Configuration & Dependency Management: This includes the environment variables for API keys (for OpenAI and Anthropic) and configuration files (system-prompts.txt, requirements.txt).
• LLM Provider Adapters: Dedicated modules (or subcomponents) for integrating with different language model providers:
o OpenAI Adapter: Handles API requests and responses for OpenAI models.
o Anthropic Adapter: Handles API interactions for Anthropic models.
o Ollama Adapter: Manages the connections to local models (with auto-download support).
• Test Engine (Orchestration Core): The central engine that coordinates:
o Iteration/Looping through test runs.
o Applying each rule against the target LLM.
o Handling firewall mode – if enabled, the module compares results against a defined “pass-condition” (e.g., the response “true”).
• Response Analyzer: Interprets the LLM’s response, checks for vulnerabilities (such as prompt stealing or distraction), and determines if the test passed or failed.
• Output Formatter: Formats results, optionally outputting a JSON file with detailed findings.
Out the Relationships and Data Flow
• The CLI parses these arguments, then passes control to the Test Engine.
• The Test Engine:
• Loads system prompts from system-prompts.txt.
• Reads and parses the chosen YAML rule files (or “all” rules from the rules/ directory).
• For each test iteration, it sends attack prompts to the target LLM by invoking the appropriate LLM Provider Adapter.
• From the CLI to the Test Engine.
• From the Test Engine to the Rule Loader.
• From the Test Engine to the LLM Provider Adapters.
• From the LLM Provider Adapters back to the Response Analyzer.
• From the Analyzer to the Output Formatter.
Architectural Patterns and Design Principles
• Separation of concerns: Notice how the scanning engine is decoupled from the rule parsing logic and the API adapters. This separation allows easy extension (for example, adding new provider interfaces or new rule types).
• Modularity: Each LLM provider has its distinct module or adapter. This modular design promotes maintainability.
• Configuration-driven behavior: The tool’s behavior is influenced by external rule files (YAML) and system prompts files, meaning that the tool is highly configurable.
Relevant Technologies and Frameworks
• Python is the main programming language.
• YAML is used for defining test rules.
• API integrations with OpenAI (REST API), Anthropic, and Ollama (possibly managed through system calls or specific libraries).
• Environment variables for sensitive configurations (API keys).