How keycloak/keycloak-benchmark works
Main Components
• Benchmark Module:
• This module (found under “benchmark”) contains load test scripts and supporting code (Java, Scala, Python scripts) that simulate user interactions and generate performance data using Gatling.
• It includes multiple sub-packages for running different test scenarios (e.g., authentication flows, administration actions, session management).
• Provisioning Module:
• Under “provision”, “ansible”, “docker-compose”, “minikube” and “openshift” folders, multiple provisioning scripts establish the necessary environment.
• These components automate the setup of cloud infrastructure (AWS scripts for Aurora, EC2, Route53, etc.), container orchestration platforms (minikube/Openshift), and sometimes additional components like Infinispan alerts.
• The CI/CD pipelines (inside the .github/workflows and .github/actions directories) integrate these provisioning tasks via GitHub Actions.
• Dataset Module:
• The “dataset” folder provides a Keycloak add-on that pre-populates realms, clients, and users to create a realistic data store for performance tests.
• Observability and Reporting:
• Parts of the provisioning and benchmark modules mention Grafana, Prometheus, and various configuration files to deploy dashboards and collect test run metrics.
• CI/CD Integration:
• GitHub Actions and workflows (found in .github/) coordinate tasks such as creating or deleting databases, managing keycloak deployments, executing benchmarks, and even running chaos and scalability benchmarks.
Relationships and Interactions
• The Benchmark Module issues test requests directly against a Keycloak instance. Its output is used for generating performance metrics.
• The Provisioning Module is responsible for setting up the test environment. It “creates” or “deletes” resources (such as Kubernetes clusters, AWS resources, or Docker containers) that host Keycloak, the observability stack, and ancillary services.
• The Dataset Module runs before or alongside the benchmark tests to ensure that a realistic set of Keycloak data (users, realms, etc.) is prepared.
• GitHub Actions and associated CI workflows orchestrate these tasks, ensuring that provisioning occurs first, then dataset preparation is complete, followed by running benchmarks and finally capturing observability data.
Architectural Patterns and Design Principles
• Separation of Concerns: The project cleanly separates load testing (benchmark), environment provisioning (provision and ansible), and test data creation (dataset). Each module handles a distinct responsibility.
• Infrastructure as Code and Automation: Provisioning scripts, ansible playbooks, and GitHub workflows reflect an automated CI/CD-driven deployment and test infrastructure.
Technologies, Frameworks, and Tools
• Benchmarking is driven by Gatling (with Scala/Java components) as seen in the “benchmark” directory.
• The provisioning aspect uses technologies such as Ansible, Docker-Compose, Helm charts, and Terraform/OpenTofu (in the “provision” directories) and touches on AWS, Openshift, and Kubernetes.
• The observability stack integrates Grafana, Prometheus, and possibly custom scripts to generate reports.
• CI/CD integration uses GitHub Actions, as shown by workflow YAML files.