# elastic/cloudbeat architecture

> Architecture diagram of the GitHub repository elastic/cloudbeat, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/elastic/cloudbeat
- Repository: https://github.com/elastic/cloudbeat

## Overview

##### Main Components / Layers
- Cloud Asset Collection/Fetchers:
– Represented by modules within internal/dataprovider and internal/inventory.
– These components include providers for AWS, Azure, GCP, and Kubernetes.
– Emphasize that each cloud provider has a “fetcher” component (e.g., awsfetcher, azurefetcher, gcpfetcher) that iterates over the cloud resources.
- Policy/Evaluation Engine:
– Components present in internal/evaluator, internal/processor, and internal/flavors.
– These modules process the raw asset data from the fetchers.
– The engine is responsible for evaluating the assets against security policies.
- Policy Definitions and Security Rules:
– Represent the “security-policies” directory.
– These files (in Rego, YAML, etc.) are used by the evaluation engine to determine compliance.
– Show generally “Policy Repository” as a separate block that feeds into the evaluation process.
- Data Transformation & Enrichment:
– Modules in internal/transformer and internal/uniqueness perform transformations, adding metadata, deduplication, or enriching events before they’re sent to Elasticsearch.
- Output & Reporting:
– The final stage is an “Output/Publisher” component (often within internal/flavors/publisher).
– This component sends findings to an Elastic Stack (Elasticsearch) instance.
– You may also note integration points with Kibana’s Cloud Security plugin for display/alerting.
- Orchestration/Launch and Configuration:
– The cmd/ and main.go are entry points, and configurations (internal/config) are used to set up the run-time behavior.
– Deployment files (deploy, tests, scripts, etc.) indicate support for multiple deployment formats (Docker, Kubernetes manifests, and cloud deployment templates).

##### Relationships and Data Flow
- Flow from the fetchers to the “Evaluation Engine”, where data is normalized, enriched, and then assessed against the “Policy Repository”.
- Configuration management (the internal/config module) connected to all major pipeline components.

##### Architectural Patterns and Principles
- Modular design: Each functional area (fetching, evaluation, transformation, publishing) is modularized in separate folders/sub-packages.
- Extensibility: The policy engine supports plugging in new cloud providers or evaluation strategies as needed.
- Separation of concerns: Clearly separate asset collection from evaluation and output.

##### Technologies and Frameworks
- Golang: Mark the language used (visible in go.mod, main.go, etc.).
- Elasticsearch: As the target for findings.
- Rego / Open Policy Agent (OPA) style policies: Indicated by the Rego files in security-policies.
- Hermit for local development/tooling (referenced in the README).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Orchestration & Configuration
    subgraph "Orchestration & Configuration"
        CMD["CMD (Launch)"]:::config
        CONFIG["Config"]:::config
        MAIN["Main"]:::config
        CMD --> CONFIG
        CONFIG --> MAIN
    end

    MAIN -->|"starts"| CP["Cloud Providers"]:::external

    subgraph "Cloud Providers"
        AWS["AWS"]:::external
        AZURE["Azure"]:::external
        GCP["GCP"]:::external
        K8S["Kubernetes"]:::external
    end

    CP --> AWS
    CP --> AZURE
    CP --> GCP
    CP --> K8S

    %% Asset Collection/Fetchers
    subgraph "Asset Collection/Fetchers"
        DP["Data Provider"]:::fetcher
        AWSF["AWS Fetcher"]:::fetcher
        AZUREF["Azure Fetcher"]:::fetcher
        GCPF["GCP Fetcher"]:::fetcher
    end

    AWS -->|"fetches-assets"| DP
    K8S -->|"fetches-assets"| DP
    AZURE -->|"fetches-assets"| AZUREF
    GCP -->|"fetches-assets"| GCPF
    DP --> AWSF
    DP --> AZUREF
    DP --> GCPF

    %% Policy/Evaluation Engine
    subgraph "Policy Evaluation Engine"
        EVAL["Evaluator"]:::policy
        PROC["Processor"]:::policy
        FLAVORS["Flavors (Asset Inventory & Benchmark)"]:::policy
    end

    AWSF -->|"sends-data"| EVAL
    AZUREF -->|"sends-data"| EVAL
    GCPF -->|"sends-data"| EVAL

    %% Policy Repository feeding Evaluation Engine
    PR["Policy Repository"]:::policy
    PR -->|"provides-policies"| EVAL

    EVAL -->|"evaluates"| PROC
    PROC --> FLAVORS

    %% Data Transformation & Enrichment
    subgraph "Data Transformation & Enrichment"
        TRANS["Transformer"]:::transform
        UNI["Uniqueness"]:::transform
    end

    FLAVORS --> TRANS
    TRANS --> UNI

    %% Output & Reporting
    PUB["Publisher"]:::output
    UNI --> PUB
    PUB -->|"pushes-findings"| ES["Elasticsearch/Kibana"]:::external

    %% External CI/CD Components
    CICD["CI/CD (.buildkite, .github, tests)"]:::external
    CICD --- MAIN

    %% Click Events
    click DP "https://github.com/elastic/cloudbeat/tree/main/internal/dataprovider"
    click AWSF "https://github.com/elastic/cloudbeat/tree/main/internal/inventory/awsfetcher"
    click AZUREF "https://github.com/elastic/cloudbeat/tree/main/internal/inventory/azurefetcher"
    click GCPF "https://github.com/elastic/cloudbeat/tree/main/internal/inventory/gcpfetcher"
    click EVAL "https://github.com/elastic/cloudbeat/tree/main/internal/evaluator"
    click PROC "https://github.com/elastic/cloudbeat/tree/main/internal/processor"
    click FLAVORS "https://github.com/elastic/cloudbeat/tree/main/internal/flavors"
    click PR "https://github.com/elastic/cloudbeat/tree/main/security-policies"
    click TRANS "https://github.com/elastic/cloudbeat/tree/main/internal/transformer"
    click UNI "https://github.com/elastic/cloudbeat/tree/main/internal/uniqueness"
    click PUB "https://github.com/elastic/cloudbeat/blob/main/internal/flavors/publisher.go"
    click CMD "https://github.com/elastic/cloudbeat/tree/main/cmd"
    click CONFIG "https://github.com/elastic/cloudbeat/tree/main/internal/config"
    click MAIN "https://github.com/elastic/cloudbeat/blob/main/main.go"

    %% Styles
    classDef config fill:#FFEBB2,stroke:#E68A00,stroke-width:2px;
    classDef fetcher fill:#FFE6E6,stroke:#CC0000,stroke-width:2px;
    classDef policy fill:#E6FFE6,stroke:#00CC66,stroke-width:2px;
    classDef transform fill:#E6E6FF,stroke:#0000CC,stroke-width:2px;
    classDef output fill:#FFF0E6,stroke:#E68A00,stroke-width:2px;
    classDef external fill:#F2F2F2,stroke:#808080,stroke-width:2px;
```
