How elastic/cloudbeat works
Main Components / Layers
• Cloud Asset Collection/Fetchers:
– Represented by modules within internal/dataprovider and internal/inventory.
– These components include providers for AWS, Azure, GCP, and Kubernetes.
– Emphasize that each cloud provider has a “fetcher” component (e.g., awsfetcher, azurefetcher, gcpfetcher) that iterates over the cloud resources.
• Policy/Evaluation Engine:
– Components present in internal/evaluator, internal/processor, and internal/flavors.
– These modules process the raw asset data from the fetchers.
– The engine is responsible for evaluating the assets against security policies.
• Policy Definitions and Security Rules:
– Represent the “security-policies” directory.
– These files (in Rego, YAML, etc.) are used by the evaluation engine to determine compliance.
– Show generally “Policy Repository” as a separate block that feeds into the evaluation process.
• Data Transformation & Enrichment:
– Modules in internal/transformer and internal/uniqueness perform transformations, adding metadata, deduplication, or enriching events before they’re sent to Elasticsearch.
• Output & Reporting:
– The final stage is an “Output/Publisher” component (often within internal/flavors/publisher).
– This component sends findings to an Elastic Stack (Elasticsearch) instance.
– You may also note integration points with Kibana’s Cloud Security plugin for display/alerting.
• Orchestration/Launch and Configuration:
– The cmd/ and main.go are entry points, and configurations (internal/config) are used to set up the run-time behavior.
– Deployment files (deploy, tests, scripts, etc.) indicate support for multiple deployment formats (Docker, Kubernetes manifests, and cloud deployment templates).
Relationships and Data Flow
• Flow from the fetchers to the “Evaluation Engine”, where data is normalized, enriched, and then assessed against the “Policy Repository”.
• Configuration management (the internal/config module) connected to all major pipeline components.
Architectural Patterns and Principles
• Modular design: Each functional area (fetching, evaluation, transformation, publishing) is modularized in separate folders/sub-packages.
• Extensibility: The policy engine supports plugging in new cloud providers or evaluation strategies as needed.
• Separation of concerns: Clearly separate asset collection from evaluation and output.
Technologies and Frameworks
• Golang: Mark the language used (visible in go.mod, main.go, etc.).
• Elasticsearch: As the target for findings.
• Rego / Open Policy Agent (OPA) style policies: Indicated by the Rego files in security-policies.
• Hermit for local development/tooling (referenced in the README).