How giskard-ai/giskard works
Purpose and package layout
Giskard v3 is a Python 3.12+ monorepo for evaluating, red-teaming, and generating tests for LLMs, agents, RAG systems, and arbitrary sync/async targets. The root distribution exposes focused namespace packages under giskard.*; giskard_pypi_shim/__init__.py supports the aggregate PyPI package. Core user-facing APIs are giskard.checks and optional giskard.scan.
Evaluation execution: Checks
giskard-checks is the primary evaluation runtime. A caller builds a Scenario of target interactions and checks, runs it asynchronously, and receives structured results/traces; Suite batches scenarios.
• libs/giskard-checks/src/giskard/checks/core/scenario.py
• libs/giskard-checks/src/giskard/checks/scenarios/runner.py
• libs/giskard-checks/src/giskard/checks/scenarios/suite.py
The target boundary is deliberately generic: user-provided sync/async callables consume inputs and return outputs, optionally with trace data. Interaction/trace abstractions normalize those executions before checks inspect them.
• libs/giskard-checks/src/giskard/checks/core/interaction/interaction.py
• libs/giskard-checks/src/giskard/checks/core/interaction/trace.py
Checks, judges, and reporting
Checks implement assertions over extracted interaction data, from deterministic built-ins to LLM-based judges. Result, extraction, testcase, and input-generation contracts isolate check authors from target invocation details.
• libs/giskard-checks/src/giskard/checks/core/check.py
• libs/giskard-checks/src/giskard/checks/core/result.py
• libs/giskard-checks/src/giskard/checks/core/extraction.py
Built-ins include composition/comparison, JSON/text/semantic checks, Rego policy, and LLM judges for groundedness, relevance, conformity, contradiction, and toxicity. Results export to JUnit or Hub-oriented formats.
• libs/giskard-checks/src/giskard/checks/builtin/composition.py
• libs/giskard-checks/src/giskard/checks/judges/judge.py
• libs/giskard-checks/src/giskard/checks/export/junit.py
Scan: automated red teaming and quality evaluation
giskard-scan composes generated scenarios into vulnerability and knowledge-base quality scans. Its public scan orchestration, catalog/registry, vulnerability types, and quality paths are the high-level entry boundary.
• libs/giskard-scan/src/giskard/scan/__init__.py
• libs/giskard-scan/src/giskard/scan/vulnerability.py
• libs/giskard-scan/src/giskard/scan/quality.py
Generators create adversarial, prompt-injection, jailbreak, and RAG-oriented test inputs, often using prompt templates and LLM generation. Generated suites flow into the Checks scenario runtime rather than requiring a separate target protocol.
• libs/giskard-scan/src/giskard/scan/generators/base.py
• libs/giskard-scan/src/giskard/scan/generators/adversarial.py
• libs/giskard-scan/src/giskard/scan/generators/knowledge_base/base.py
LLM and workflow substrate
giskard-llm provides provider-neutral chat/response/embedding types, routing, retries, and provider-specific payload translators. It integrates with OpenAI, Anthropic, Google, Azure OpenAI, and Azure AI SDKs, selected through optional dependencies and credentials.
• libs/giskard-llm/src/giskard/llm/routing.py
• libs/giskard-llm/src/giskard/llm/providers/base.py
• libs/giskard-llm/src/giskard/llm/translators/
giskard-agents adds generator adapters, prompt templating, tools, and workflow/pipeline orchestration used by LLM judges and scan generators.
• libs/giskard-agents/src/giskard/agents/generators/base.py
• libs/giskard-agents/src/giskard/agents/workflow.py
• libs/giskard-agents/src/giskard/agents/templates/prompts_manager.py
Shared cross-cutting services and integrations
giskard-core centralizes shared error/discriminated-type utilities, rate limiting, and opt-out telemetry. Telemetry is aggregated and controlled before import via environment variables.
• libs/giskard-core/src/giskard/core/telemetry/telemetry.py
• libs/giskard-core/src/giskard/core/rate_limiter/base.py
Scan also adapts external red-team ecosystems—DeepTeam, Garak, and LIDAR—behind integration bridges/adapters, preserving Giskard’s scenario/check model as the internal boundary.
• libs/giskard-scan/src/giskard/scan/integrations/deepteam/_bridge.py
• libs/giskard-scan/src/giskard/scan/integrations/garak/_bridge.py
• libs/giskard-scan/src/giskard/scan/integrations/lidar/_adapter.py
Main entry points: scenario.py Scenario · __init__.py Scan API and catalog · runner.py Scenario runner · check.py Check contracts
Components 21
Checks runtime
Async scenario evaluation and reporting.
- Scenario · Defines target interactions and checks.
libs/giskard-checks/src/giskard/checks/core/scenario.py - Suite · Batches scenario execution.
libs/giskard-checks/src/giskard/checks/scenarios/suite.py - Scenario runner · Invokes targets and coordinates checks.
libs/giskard-checks/src/giskard/checks/scenarios/runner.py - Interaction model · Normalizes target inputs and outputs.
libs/giskard-checks/src/giskard/checks/core/interaction/interaction.py - Trace model · Captures optional execution trace data.
libs/giskard-checks/src/giskard/checks/core/interaction/trace.py - Check contracts · Isolates check logic from target invocation.
libs/giskard-checks/src/giskard/checks/core/check.py - Results and extraction · Structured outcomes and extracted interaction data.
libs/giskard-checks/src/giskard/checks/core/result.py - Built-in checks · Composition, text, JSON, semantic, and policy checks.
libs/giskard-checks/src/giskard/checks/builtin/composition.py - LLM judges · Evaluates groundedness, relevance, safety, and conformity.
libs/giskard-checks/src/giskard/checks/judges/judge.py - JUnit export · Exports evaluation results.
libs/giskard-checks/src/giskard/checks/export/junit.py
Scan
Automated red teaming and quality scans.
- Scan API and catalog · Entry point for vulnerability and quality scans.
libs/giskard-scan/src/giskard/scan/__init__.py - Scan generators · Builds adversarial and prompt-injection scenarios.
libs/giskard-scan/src/giskard/scan/generators/adversarial.py - Knowledge-base generators · Creates RAG-oriented quality test inputs.
libs/giskard-scan/src/giskard/scan/generators/knowledge_base/base.py - External red-team bridges · Adapts DeepTeam, Garak, and LIDAR to internal scenarios.
libs/giskard-scan/src/giskard/scan/integrations/deepteam/_bridge.py
LLM workflow substrate
Provider-neutral LLM execution and agent workflows.
- Agent workflows · Composes generators, tools, and pipelines.
libs/giskard-agents/src/giskard/agents/workflow.py - Prompt templates · Manages prompts used by generators and judges.
libs/giskard-agents/src/giskard/agents/templates/prompts_manager.py - LLM routing · Routes requests with retries to configured providers.
libs/giskard-llm/src/giskard/llm/routing.py - Provider adapters · Translates neutral requests to provider payloads.
libs/giskard-llm/src/giskard/llm/providers/base.py
Shared services
Cross-cutting runtime controls.
- Telemetry · Environment-controlled aggregated telemetry.
libs/giskard-core/src/giskard/core/telemetry/telemetry.py - Rate limiter · Coordinates request rate limits.
libs/giskard-core/src/giskard/core/rate_limiter/base.py
Other components
- User target · Consumes inputs and may return trace data.
Connections 23
- Suite → Scenario: batches
- Scenario → Scenario runner: runs through
- Scenario runner → User target: invokes
- Scenario runner → Interaction model: normalizes
- Interaction model → Trace model: includes optional
- Scenario runner → Check contracts: evaluates
- Check contracts → Built-in checks: implemented by
- Check contracts → LLM judges: implemented by
- Check contracts → Results and extraction: produces
- Results and extraction → JUnit export: exports as
- Scan API and catalog → Scan generators: orchestrates
- Scan API and catalog → Knowledge-base generators: orchestrates
- Scan API and catalog → External red-team bridges: uses
- Scan generators → Scenario: generates
- Knowledge-base generators → Scenario: generates
- External red-team bridges → Scenario: adapts into
- Scan generators → Agent workflows: uses
- LLM judges → Agent workflows: uses
- Agent workflows → Prompt templates: renders
- Agent workflows → LLM routing: requests LLMs through
- LLM routing → Provider adapters: dispatches to
- LLM routing → Rate limiter: is constrained by
- Scan API and catalog → Telemetry: reports via