mizcausevic-dev/prompt-injection-bench

Purpose and principal workflow: prompt-injection-bench is a local corpus and heuristic harness for regression-testing an AI agent against synthetic prompt-injection cases. The external operator starts the main workflow by invoking pib run; the target agent is exercised separately in the intended trust boundary, and the harness evaluates its responses from a complete JSONL transcript. Echo and blanket-refusal adapters are local diagnostics, not connections to a live agent. Reports are heuristic signals, not safety determinations.

Components

Command-line and corpus

• pib CLI — src/pib/cli.py: Implements list, validate, build-jsonl, and run; selects a run adapter, invokes the harness, prints text or JSON, and applies an optional pass-rate threshold.

• Attack-case corpus — corpus/DEL-001.json: A representative record in the 30-case, 10-category synthetic corpus; records carry attack prompts, expected safe behavior, scoring rules, and Agent Card refusal-category back-references.

• Corpus loader and exporter — src/pib/loader.py: Loads and validates individual corpus JSON files or flat JSONL records, checks cross-record IDs and category prefixes, and writes corpus JSONL.

• Record and report schemas — src/pib/schema.py: Defines Pydantic models and constrained fields for attack records, individual run results, and aggregate reports.

• Attack taxonomy and scoring guidance — TAXONOMY.md: Documents the attack categories, scoring methods, Agent Card cross-references, and severity rubric.

Run and evaluation

• Run orchestration and adapters — src/pib/runner.py: Iterates through records, obtains each response and tool-attempt boolean from an adapter, invokes scoring, aggregates category/severity rates and critical failures, and renders text reports; it also implements transcript, echo, and refusal adapters.

• Response scorer — src/pib/scorer.py: Applies the record’s selected heuristic to response text and, for tool-invocation cases, requires an observed boolean; returns a per-case result.

• Operator — *(external; null repository path)*: Invokes the CLI and supplies run options, including the transcript path when evaluating recorded responses.

• Target agent — *(external; null repository path)*: Receives the synthetic cases through the operator’s chosen trust boundary and produces responses; this repository does not call the agent directly.

• Run transcript — *(external input; null repository path)*: The operator-provided JSONL of responses and observed tool-attempt booleans, consumed by the transcript adapter.

Relationships

• Operator -> pib CLI: invokes pib with a command and options; the executable entry point is declared in pyproject.toml and command handling is implemented in src/pib/cli.py.

• Target agent -> Run transcript: produces responses and observed tool-attempt information during the operator’s separate test run; the README specifies this transcript format and does not establish an automated agent connection.

• Operator -> Run transcript: supplies a complete transcript for --adapter transcript; pib requires a response and explicit boolean for every corpus ID.

• Attack-case corpus -> Corpus loader and exporter: provides JSON records that the loader discovers and parses; the loader also supports flat JSONL input.

• Corpus loader and exporter -> Record and report schemas: validates loaded attack records using the Pydantic attack-record model.

• pib CLI -> Corpus loader and exporter: loads records for commands and uses the exporter for build-jsonl.

• pib CLI -> Run orchestration and adapters: selects an adapter and calls run_corpus for a run; echo, refusal, and transcript are alternative adapter choices, not sequential stages.

• Run transcript -> Run orchestration and adapters: the transcript adapter reads JSONL, rejects malformed, duplicate, unknown, or missing IDs, and looks up each response and tool-attempt boolean.

• Run orchestration and adapters -> Response scorer: calls score_response once per record and adapter response.

• Response scorer -> Record and report schemas: constructs a per-case result; the runner uses those results to construct an aggregate report.

• Run orchestration and adapters -> Record and report schemas: creates the aggregate report with target label, pass rates, critical failures, and results.

• pib CLI -> Run orchestration and adapters: renders the returned report as text or JSON and applies the optional --fail-under exit gate.

Coverage limits: The sampled sources establish the CLI-to-loader-to-runner-to-scorer flow and transcript behavior. The corpus and taxonomy are represented by tree/README evidence rather than sampled record contents or taxonomy source. No live-agent API, tool-trace capture mechanism, or hosted visual-harness implementation is established here; the README identifies the visual harness as a separate deployment.