# kube-logging/logging-operator architecture

> Architecture diagram of the GitHub repository kube-logging/logging-operator, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kube-logging/logging-operator
- Repository: https://github.com/kube-logging/logging-operator

## Overview

##### Project Type and Purpose
- This is a Kubernetes Operator (an “open-source tool”) implemented in Go.
- It automates deployment and configuration of a logging pipeline on Kubernetes:
- Fluent Bit DaemonSet as log collectors on each node
- Fluentd or syslog-ng as log forwarders
- CustomResourceDefinitions (CRDs) that drive configuration: Logging, Flow, Output, ClusterFlow, ClusterOutput, and their syslog-ng equivalents.

##### File Structure
- cmd, main.go: the entrypoint for the operator binary (controller manager).
- controllers/*: controller implementations for Logging, Flows, Outputs, Extensions, Telemetry.
- pkg/resources/*: helper code that “reconciles” CRs into Kubernetes primitives (DaemonSets, StatefulSets, Services, RBAC).
- pkg/sdk/*: generated CRD types and client code for extensions and logging APIs.
- config/: Kubernetes manifests & Kustomize overlays for CRDs, RBAC, webhook, manager deployment.
- charts/: Helm charts to install the operator and sample logging pipelines.
- images/: sidecar / utility images (config-reloader, drain-watch, node-exporter).
- e2e/: end-to-end tests deploying operator into a Kind cluster.

##### README
- High-level flow:
1. Operator watches CRDs.
2. Operator reconciles to deploy Fluent Bit collectors and Fluentd/syslog-ng forwarders.
3. Fluent Bit reads logs, enriches with metadata from the API.
4. Forwarder filters/transforms per Flow CRs and ships to Outputs (Elasticsearch, S3, Loki, etc.).
- Architecture section confirms components: log collectors, log forwarders, flows, outputs, control-plane operator.

##### Main Components
a. Kubernetes Control Plane / API Server
– CRDs: Logging, Flow, Output, ClusterFlow, ClusterOutput, SyslogNG equivalents
– Operator’s Deployment (controller-manager) and Webhook Server
b. Operator (Control Plane)
– main.go → manager (controller-runtime)
– controllers/*
– pkg/resources for reconciliation logic
– config-reloader sidecar (Hot-reload)
c. Data Plane
– Fluent Bit DaemonSet on every node (collect logs + metadata enrichment via K8s API)
– Fluentd / syslog-ng Forwarder (StatefulSet or DaemonSet)
– Telemetry-controller (optional)
d. External Services (Outputs)
– Elasticsearch, S3, GCS, Loki, Datadog, etc.
– (show multiple, configurable per Output CR)
e. Supporting Infrastructure
– Prometheus metrics & ServiceMonitors
– RBAC resources (Roles, RoleBindings)
– Webhook Admission for validation
– Helm Chart / Kustomize as install layer

##### Relationships and Data Flows
- Operator watches CRDs → reconciles → deploys/updates Collector and Forwarder workloads.
- Fluent Bit reads logs from node’s file system → queries API Server for Pod metadata → forwards enriched logs to Forwarder.
- Forwarder applies Flow rules (filters, transforms) → forwards to configured Outputs over TLS.
- Metrics scraped by Prometheus via ServiceMonitor.

##### Architectural Patterns and Design Principles
- Kubernetes Operator pattern (control loop + reconciliation)
- Use of Controller-Runtime library in Go
- CRD-driven configuration → declarative infrastructure
- Sidecar pattern for dynamic configuration reload (config-reloader)
- Helm + Kustomize for deployment templating
- Modular code: separate pkg/resources per component (fluentbit, fluentd, syslog-ng, eventtailer, nodeagent)

##### Essential Architectural Elements
- CRD definitions as the “source of truth”
- Controller Manager (multi-controller binary)
- Admission Webhook (validation)
- Dynamic reconfiguration via config-reloader sidecar
- Node-level DaemonSets vs. clustered StatefulSets
- Secure communication channels (TLS between components and outputs)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Control Plane Layer
    subgraph "Control Plane" 
        direction TB
        APIServer["Kubernetes API Server"]:::infra
        CRDs["CRD Definitions"]:::control
        OperatorDeployment["Operator Deployment (controller-manager)"]:::control
        AdmissionWebhook["Admission Webhook Server"]:::control
        ControllerManager["Controller Manager\n(main.go)"]:::control
        Controllers["Controllers\n(logging, flows, outputs, extensions, telemetry)"]:::control
        Reconciler["Reconciliation Helpers\n(pkg/resources)"]:::control
        ConfigReloader["Config-Reloader Sidecar"]:::control
    end

    %% Data Plane - Collectors
    subgraph "Data Plane - Collectors"
        direction TB
        FluentBit["Fluent Bit DaemonSet"]:::collector
    end

    %% Data Plane - Forwarders
    subgraph "Data Plane - Forwarders"
        direction TB
        Fluentd["Fluentd Forwarder"]:::forwarder
        SyslogNG["Syslog-ng Forwarder"]:::forwarder
    end

    %% External Outputs
    subgraph "External Outputs"
        direction TB
        Elasticsearch["Elasticsearch"]:::external
        S3["S3"]:::external
        Loki["Loki"]:::external
        Datadog["Datadog"]:::external
    end

    %% Supporting Infrastructure
    subgraph "Supporting Infrastructure"
        direction TB
        Prometheus["Prometheus & ServiceMonitor"]:::infra
        RBAC["RBAC Resources"]:::infra
        HelmKustomize["Helm Chart / Kustomize"]:::infra
    end

    %% Connections
    APIServer -->|"watches CRDs"| OperatorDeployment
    CRDs --> APIServer
    OperatorDeployment --> ControllerManager
    ControllerManager --> Controllers
    Controllers --> Reconciler
    ControllerManager --> AdmissionWebhook
    ControllerManager --> ConfigReloader

    OperatorDeployment -->|"reconcile"| FluentBit
    OperatorDeployment -->|"reconcile"| Fluentd
    OperatorDeployment -->|"reconcile"| SyslogNG

    FluentBit -->|"gets metadata"| APIServer
    FluentBit -->|"forwards logs"| Fluentd
    FluentBit -->|"forwards logs"| SyslogNG

    Fluentd -->|"ships logs over TLS"| Elasticsearch
    Fluentd -->|"ships logs over TLS"| S3
    Fluentd -->|"ships logs over TLS"| Loki
    Fluentd -->|"ships logs over TLS"| Datadog

    SyslogNG -->|"ships logs over TLS"| Elasticsearch
    SyslogNG -->|"ships logs over TLS"| S3
    SyslogNG -->|"ships logs over TLS"| Loki
    SyslogNG -->|"ships logs over TLS"| Datadog

    Prometheus -->|"scrape metrics"| FluentBit
    Prometheus -->|"scrape metrics"| Fluentd
    Prometheus -->|"scrape metrics"| SyslogNG
    Prometheus -->|"scrape metrics"| OperatorDeployment

    HelmKustomize --> OperatorDeployment
    HelmKustomize --> RBAC
    RBAC --> OperatorDeployment

    %% Click Events for component mapping
    click CRDs "https://github.com/kube-logging/logging-operator/tree/master/config/crd/bases"
    click CRDs "https://github.com/kube-logging/logging-operator/tree/master/charts/logging-operator/crds"
    click CRDs "https://github.com/kube-logging/logging-operator/tree/master/pkg/sdk/logging/api/v1alpha1"
    click CRDs "https://github.com/kube-logging/logging-operator/tree/master/pkg/sdk/logging/api/v1beta1"

    click OperatorDeployment "https://github.com/kube-logging/logging-operator/blob/master/config/default/manager.yaml"
    click OperatorDeployment "https://github.com/kube-logging/logging-operator/blob/master/config/manager/manager.yaml"
    click OperatorDeployment "https://github.com/kube-logging/logging-operator/blob/master/charts/logging-operator/templates/deployment.yaml"

    click AdmissionWebhook "https://github.com/kube-logging/logging-operator/tree/master/pkg/webhook/podhandler"
    click AdmissionWebhook "https://github.com/kube-logging/logging-operator/blob/master/config/webhook/service.yaml"
    click AdmissionWebhook "https://github.com/kube-logging/logging-operator/blob/master/config/webhook/kustomization.yaml"

    click ControllerManager "https://github.com/kube-logging/logging-operator/blob/master/main.go"
    click ControllerManager "https://github.com/kube-logging/logging-operator/blob/master/cmd/docs.go"

    click Controllers "https://github.com/kube-logging/logging-operator/blob/master/controllers/logging/logging_controller.go"
    click Controllers "https://github.com/kube-logging/logging-operator/blob/master/controllers/logging/loggingroute_controller.go"
    click Controllers "https://github.com/kube-logging/logging-operator/blob/master/controllers/logging/axosyslog_controller.go"
    click Controllers "https://github.com/kube-logging/logging-operator/blob/master/controllers/logging/telemetry_controller_controller.go"
    click Controllers "https://github.com/kube-logging/logging-operator/blob/master/controllers/extensions/eventtailer_controller.go"
    click Controllers "https://github.com/kube-logging/logging-operator/blob/master/controllers/extensions/hosttailer_controller.go"

    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/fluentbit/daemonset.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/fluentbit/config.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/fluentbit/rbac.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/fluentbit/service.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/fluentbit/prometheusrules.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/fluentd/statefulset.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/fluentd/config.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/fluentd/outputsecret.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/fluentd/pdb.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/fluentd/service.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/fluentd/prometheusrules.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/syslogng/statefulset.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/syslogng/configcheck.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/syslogng/configsecret.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/syslogng/prometheusrules.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/eventtailer/statefulset.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/eventtailer/configmap.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/eventtailer/serviceaccount.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/nodeagent/daemonset.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/nodeagent/config.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/nodeagent/rbac.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/nodeagent/service.go"
    click Reconciler "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/telemetry-controller/resources.go"

    click ConfigReloader "https://github.com/kube-logging/logging-operator/blob/master/images/config-reloader/cmd/configreloader/main.go"
    click ConfigReloader "https://github.com/kube-logging/logging-operator/blob/master/images/config-reloader/pkg/configreloader/configreloader.go"

    click FluentBit "https://github.com/kube-logging/logging-operator/blob/master/charts/logging-operator/templates/logging/fluentbit.yaml"
    click Fluentd "https://github.com/kube-logging/logging-operator/blob/master/charts/logging-operator/templates/logging/eventtailer.yaml"
    click SyslogNG "https://github.com/kube-logging/logging-operator/blob/master/pkg/resources/syslogng/statefulset.go"

    click Prometheus "https://github.com/kube-logging/logging-operator/blob/master/charts/logging-operator/templates/service_monitor.yaml"
    click RBAC "https://github.com/kube-logging/logging-operator/tree/master/config/rbac/"

    click HelmKustomize "https://github.com/kube-logging/logging-operator/tree/master/charts/logging-operator"
    click HelmKustomize "https://github.com/kube-logging/logging-operator/tree/master/charts/logging-demo"

    %% Styles
    classDef control fill:#D0E7FF,stroke:#3178C6
    classDef collector fill:#E6FDD0,stroke:#4CAF50
    classDef forwarder fill:#FFE5C0,stroke:#FB8C00
    classDef external fill:#E5D0FF,stroke:#9C27B0
    classDef infra fill:#F0F0F0,stroke:#666666
    class CRDs,OperatorDeployment,AdmissionWebhook,ControllerManager,Controllers,Reconciler,ConfigReloader control
    class FluentBit collector
    class Fluentd,SyslogNG forwarder
    class Elasticsearch,S3,Loki,Datadog external
    class APIServer,Prometheus,RBAC,HelmKustomize infra
```
