How fbsamples/caldera-security-tests works
Project type and purpose
• This is a Go-based command-line application (“caldera-security-tests”) whose sole purpose is to automate security regression tests (TTP Runner) against a live MITRE CALDERA instance.
• It is designed to be run locally or inside a CI workflow (GitHub Actions) to routinely validate that known vulnerabilities remain patched.
File structure
Top-level directories and files:
• cmd/
– Contains subcommands implemented in Go (testEnv, storedXSSUno, Dos, Tres, config, logging). These map directly to the CLI’s functionality.
• main.go
– Instantiates the root Cobra command and wire-up of subcommands.
• cmd/config/config.yaml
– Default configuration file template (connection info for CALDERA, Docker parameters, etc.).
• magefiles/
– Build and release automation via Mage (compiling, testing, docs generation).
• .github/workflows/
– CI pipelines: SRP (security regression pipeline), semgrep, tests, goreleaser, etc.
• docs/ & images/
• go.mod, go.sum, .goreleaser.yml
– Dependency and release orchestration.
Extract architectural insights from the README
• The CLI orchestrates two phases:
a) testEnv → spin up or tear down an ephemeral CALDERA test environment (e.g., Docker).
b) storedXSS… → execute one of three automated attack TTPs against the running environment via the CALDERA API.
• Run regularly in a GitHub Actions workflow (srp.yaml) to gate merge commits and open issues on regression.
a. Main components
1) CLI Application (Go binary)
– Subcomponents (modules/packages):
• Root/Cobra & Subcommands (testEnv, storedXSSUno/Dos/Tres, config, logging)
• Config Loader (reads config.yaml)
• Docker/Environment Orchestrator (provisions CALDERA environment)
• TTP Runner (executes stored XSS attacks via HTTP API)
• Logger (structured output to stdout/stderr)
2) External CALDERA Service
– Lives in Docker or remote host; exposes REST API for plugins/TTPs.
3) GitHub Actions CI
– Triggers the CLI periodically or on PR/push; monitors exit codes; creates issues on failures.
4) Build & Release (Mage + goreleaser)
– Automates binary builds, cross-compilation, and GitHub releases.
b. Relationships and interactions
– The CLI’s testEnv command calls into the Docker/Environment Orchestrator to bring up or tear down containers running MITRE CALDERA.
– The storedXSS… commands load TTP definitions and invoke CALDERA’s REST API to run each attack; they consume API responses to detect success/failure.
– The Config Loader feeds connection endpoints and credentials into both Env Orchestrator and TTP Runner.
– GitHub Actions workflows invoke the CLI (via “uses: fbsamples/caldera-security-tests…”) on schedule or events, parse its exit code, and create issues on regressions.
– Magefiles provide a separate build pipeline: compiling the CLI, running unit tests, linting, and packaging.
c. Architectural patterns and principles
– Modular CLI design using Cobra: each command is isolated in its own package.
– Single-responsibility: Env orchestration, TTP execution, config loading, and logging are separate modules.
– Infrastructure as code potential: testEnv could be extended with Terraform or Docker Compose.
– CI/CD integration: clear separation between build (Mage + goreleaser) and test runs (GitHub Actions).
d. Key technologies and libraries
– Go (compiler, modules)
– spf13/cobra (CLI framework)
– YAML (config, GitHub workflows)
– Docker (ephemeral CALDERA environment)
– GitHub Actions (CI workflows)
– Mage + goreleaser (build & release)
– Semgrep, pre-commit hooks (QA gates)