How microsoft/rampart works
RAMPART is a pytest-native framework for testing agentic AI safety and security. Developers define attack or probe executions against an agent adapter; prompt drivers supply requests, execution strategies manage sessions and turns, evaluators assess behavior, and results capture verdicts and evidence for reporting. Supporting capabilities include adversarial payload generation, injection surfaces, LLM-backed judging, and pytest integration. Edges reflect the README and SOURCE INDEX references or shown execution calls; unsampled subsystems and their internal wiring are represented conservatively.
Main entry points: execution.py Execution lifecycle · _xpia.py XPIA strategy · evaluator.py Evaluator composition · llm_bridge.py PyRIT bridge
Components 34
Test authoring
- Public test API
rampart/__init__.py - Attack factory
rampart/attacks/_factory.py - XPIA strategy
rampart/attacks/_xpia.py - Probe factory
rampart/probes/_factory.py - Single-turn strategy
rampart/probes/_single_turn.py
Execution runtime
- Execution lifecycle
rampart/core/execution.py - Agent adapter
rampart/core/adapter.py - Prompt driver protocol
rampart/core/prompt_driver.py - LLM prompt driver
rampart/drivers/llm.py - Static prompt driver
rampart/drivers/static.py - PyRIT bridge
rampart/pyrit_bridge/llm_bridge.py - LLM configuration
rampart/core/llm.py - Prompt templates
rampart/common/templates.py
Evaluation and results
- Evaluator composition
rampart/core/evaluator.py - LLM judge
rampart/evaluators/llm_judge.py - Response matcher
rampart/evaluators/response_contains.py - Verdict and evidence
rampart/core/result.py - Shared domain types
rampart/core/types.py - Text safety
rampart/common/text.py
Payloads and surfaces
- Payload facade
rampart/payloads/_facade.py - Payload generator
rampart/payloads/_generator.py - Payload templates
rampart/payloads/template.py - Payload store
rampart/payloads/_store.py - Injection contract
rampart/core/injection.py - OneDrive surface
rampart/surfaces/onedrive.py
Pytest and reporting
- Test collection
rampart/pytest_plugin/_collection.py - pytest plugin
rampart/pytest_plugin/plugin.py - Session handling
rampart/pytest_plugin/_session.py - Report sink
rampart/reporting/sink.py - JSON report
rampart/reporting/json_file.py
Other components
- Test developer
- pytest runner
- Agent under test
- LLM service
Connections 36
- Test developer → Public test API: defines tests
- pytest runner → pytest plugin: loads plugin
- Public test API → Attack factory: exposes attacks
rampart/__init__.py - Public test API → Probe factory: exposes probes
rampart/__init__.py - Attack factory → XPIA strategy: builds strategy
rampart/attacks/_factory.py - Probe factory → Single-turn strategy: builds strategy
rampart/probes/_factory.py - XPIA strategy → Execution lifecycle: extends lifecycle
rampart/attacks/_xpia.py - Single-turn strategy → Execution lifecycle: extends lifecycle
rampart/probes/_single_turn.py - Execution lifecycle → Agent adapter: creates session
rampart/probes/_single_turn.py - Execution lifecycle → Prompt driver protocol: requests prompts
rampart/probes/_single_turn.py - Execution lifecycle → Evaluator composition: evaluates turns
rampart/core/execution.py - Execution lifecycle → Verdict and evidence: builds result
rampart/attacks/_xpia.py - Single-turn strategy → Verdict and evidence: builds result
rampart/probes/_single_turn.py - Agent under test → Agent adapter: is wrapped by
- LLM prompt driver → Prompt driver protocol: implements
rampart/drivers/llm.py - Static prompt driver → Prompt driver protocol: implements
rampart/drivers/static.py - LLM prompt driver → PyRIT bridge: sends prompts
rampart/drivers/llm.py - LLM configuration → PyRIT bridge: configures target
rampart/pyrit_bridge/llm_bridge.py - PyRIT bridge → LLM service: calls model
- LLM judge → PyRIT bridge: sends judge request
rampart/evaluators/llm_judge.py - Evaluator composition → LLM judge: composes evaluator
rampart/evaluators/llm_judge.py - Evaluator composition → Response matcher: composes evaluator
rampart/evaluators/response_contains.py - Payload facade → Payload generator: delegates generation
rampart/payloads/_facade.py - Payload generator → Payload templates: uses template
rampart/payloads/_generator.py - XPIA strategy → Injection contract: activates handles
rampart/attacks/_xpia.py - OneDrive surface → Injection contract: implements contract
rampart/surfaces/onedrive.py - pytest plugin → Test collection: collects tests
- Test collection → Execution lifecycle: invokes execution
rampart/pytest_plugin/_collection.py - Session handling → Test collection: uses collection
rampart/pytest_plugin/_session.py - Session handling → Report sink: writes results
rampart/pytest_plugin/_session.py - Report sink → JSON report: writes JSON
rampart/reporting/json_file.py - JSON report → Verdict and evidence: serializes verdict
rampart/reporting/json_file.py - XPIA strategy → Shared domain types: uses domain types
rampart/attacks/_xpia.py - Evaluator composition → Shared domain types: uses evaluation types
rampart/core/evaluator.py - LLM prompt driver → Prompt templates: renders prompt
rampart/drivers/llm.py - Verdict and evidence → Text safety: sanitizes text
rampart/core/result.py