# aws/aws-for-fluent-bit architecture

> Architecture diagram of the GitHub repository aws/aws-for-fluent-bit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aws/aws-for-fluent-bit
- Repository: https://github.com/aws/aws-for-fluent-bit

## Overview

1. **What this repository is**
   - This is the build and release repo for the AWS for Fluent Bit container images, not the upstream Fluent Bit runtime itself.
   - Its main output is versioned Linux and Windows container images that bundle Fluent Bit plus AWS-specific output plugins and supporting tooling.

2. **Primary runtime boundary**
   - The core runtime is Fluent Bit running inside a container image.
   - The image layers in AWS plugins, default configs, and in some variants an init process for multi-config startup.
   - The repo’s architecture is therefore centered on producing immutable image artifacts rather than a long-running application service.

3. **Bundled AWS plugin subsystem**
   - The key functional extension is the set of AWS output plugins: CloudWatch Logs, Kinesis Streams, and Kinesis Firehose.
   - These are treated as separately developed components that are compiled and packaged into the final image.
   - The repo supports swapping in forked plugin sources for development, which shows the plugins are a distinct subsystem from the image assembly pipeline.

4. **Image build and packaging pipeline**
   - `Makefile`, `scripts/`, and `scripts/dockerfiles/` define the build graph for release, dev, debug, init, and plugin-only images.
   - Dockerfiles are split by purpose: common build stages, runtime stages, debug/valgrind variants, and Windows-specific images.
   - Build inputs come from `linux.version`, `windows.versions`, and the version marker files in the repo root, which control what gets assembled into each image tag.

5. **Linux vs Windows product lines**
   - Linux images and Windows images are handled as separate release tracks with different rebuild policies and tagging rules.
   - Linux uses Amazon Linux base images and supports amd64/arm64 multi-arch tags.
   - Windows uses Server Core base images, publishes separate Windows tags, and rebuilds monthly to absorb Microsoft security updates.

6. **Init-process variant**
   - The `init/` subtree implements an init process used by “init” image tags.
   - This variant exists to support multi-config startup behavior, especially for FireLens-style deployments where a wrapper process prepares Fluent Bit before handing off to it.
   - Architecturally, this is a small but important startup boundary between the container entrypoint and Fluent Bit itself.

7. **Configuration and example topology**
   - Root-level configs, `ecs/`, `ecs_windows_forward_daemon/`, `examples/`, `use_cases/`, and `tests/` contain deployment recipes rather than application logic.
   - These configs show the intended data flow: application logs enter Fluent Bit via forward, tail, stdout, or socket-style inputs; filters transform records; outputs ship them to AWS destinations.
   - The `filters/replace_dots.lua` script and parser configs indicate that log enrichment and normalization are first-class concerns.

8. **AWS destination pipelines**
   - The main integration targets are CloudWatch Logs, Kinesis, Firehose, and S3.
   - The repo’s sample and ECS/Windows configs are organized around these destinations, so the architecture is effectively “input → filter → AWS output.”
   - S3 is also used as a validation and cleanup target in testing and load-test workflows, not only as a production sink.

9. **Testing hierarchy**
   - The repo has three major test layers: unit/small tests, integration tests, and load/stability tests.
   - `integ/` runs short end-to-end checks against real AWS resources or containerized validators.
   - `load_tests/` creates test infrastructure and validates throughput/log-loss thresholds.
   - `tests/` contains local compose-based and Lua filter tests for faster feedback.

10. **Test infrastructure provisioning**
   - The repo provisions AWS resources through CDK-style app folders under `integ/test_init/` and `load_tests/create_testing_resources/`.
   - These helpers create the external environment the image needs to prove correctness against real AWS services.
   - This means the system boundary is not just container runtime vs source code; it also includes cloud resources required for verification.

11. **Release and publishing pipeline**
   - The GitHub Actions workflows and `buildspec*.yml` files drive release initiation, title linting, image publishing, and sync jobs.
   - Publishing targets include Docker Hub, Amazon ECR, public ECR, and SSM public parameters.
   - The repo’s release process is tightly coupled to image tagging and version propagation across registries and parameter stores.

12. **Operational support and diagnostics**
   - `docs/`, `troubleshooting/`, and debug/valgrind Dockerfiles are part of the product architecture because they shape how the image is operated and investigated.
   - Debug variants exist specifically for crash analysis and memory tooling, which indicates a deliberate split between production runtime images and diagnostic images.

13. **Windows deployment support**
   - `ecs_windows_forward_daemon/` and the Windows-specific integration/config folders provide curated configs for ECS daemon-style forwarding on Windows.
   - This is a separate deployment pattern from Linux FireLens-style usage and reflects a distinct runtime environment and networking model.

14. **Overall system picture**
   - At a high level, the repo builds a family of container images that convert application logs into AWS logging/streaming/storage services.
   - The major architectural boundaries are: upstream Fluent Bit source, AWS plugin code, image assembly/runtime layers, cloud test infrastructure, and release/publishing infrastructure.

Main entry points: [`windows.versions`](https://github.com/aws/aws-for-fluent-bit/blob/mainline/windows.versions) (Windows Line), [`scripts/dockerfiles/runtime`](https://github.com/aws/aws-for-fluent-bit/tree/mainline/scripts/dockerfiles/runtime) (Runtime Dockerfiles), [`Makefile`](https://github.com/aws/aws-for-fluent-bit/tree/mainline/Makefile) (Build Scripts), [`integ`](https://github.com/aws/aws-for-fluent-bit/tree/mainline/integ) (Integration Tests)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_runtime["Image Runtime"]
  node_node_fluent_bit_image["Fluent Bit Image<br/>container runtime"]
  node_node_linux_line["Linux Line<br/>image track<br/>[linux.version]"]
  node_node_windows_line["Windows Line<br/>image track<br/>[windows.versions]"]
  node_node_init_variant["Init Variant<br/>startup layer"]
  node_node_runtime_common["Runtime Dockerfiles<br/>image assembly"]
  node_node_init_entrypoint["Init Process<br/>go entrypoint"]
end

subgraph group_group_plugins["AWS Plugins"]
  node_node_plugin_bundle["AWS Plugins<br/>plugin subsystem"]
  node_node_plugin_build["Plugin Build<br/>plugin compile<br/>[build_plugins.sh]"]
end

subgraph group_group_build["Build & Release"]
  node_node_build_common["Build Dockerfiles<br/>compile stages"]
  node_node_make_scripts["Build Scripts<br/>orchestration"]
  node_node_windows_build["Windows Builders<br/>orchestration"]
  node_node_release_flow["Release Flow<br/>publishing"]
end

subgraph group_group_configs["Configs & Examples"]
  node_node_configs["Core Configs<br/>fluent-bit configs<br/>[fluent-bit.conf]"]
  node_node_ecs_recipes["ECS Recipes<br/>deployment configs"]
  node_node_windows_recipes["Windows Recipes<br/>deployment configs"]
  node_node_examples["Examples<br/>sample configs"]
end

subgraph group_group_tests["Verification"]
  node_node_integration_tests["Integration Tests<br/>end-to-end"]
  node_node_load_tests["Load Tests<br/>throughput tests"]
  node_node_local_tests["Local Tests<br/>fast tests"]
end

subgraph group_group_support["Support"]
  node_node_troubleshooting["Troubleshooting<br/>diagnostics"]
end

node_node_fluent_bit_image -->|"packages"| node_node_linux_line
node_node_fluent_bit_image -->|"packages"| node_node_windows_line
node_node_fluent_bit_image -->|"includes"| node_node_init_variant
node_node_linux_line -->|"builds from"| node_node_runtime_common
node_node_windows_line -->|"builds from"| node_node_windows_build
node_node_runtime_common -->|"assembles"| node_node_fluent_bit_image
node_node_build_common -->|"feeds"| node_node_runtime_common
node_node_make_scripts -->|"drives"| node_node_build_common
node_node_make_scripts -->|"drives"| node_node_plugin_build
node_node_windows_build -->|"publishes"| node_node_release_flow
node_node_release_flow -->|"publishes"| node_node_fluent_bit_image
node_node_plugin_build -->|"produces"| node_node_plugin_bundle
node_node_plugin_bundle -->|"embedded in"| node_node_fluent_bit_image
node_node_configs -->|"configures"| node_node_fluent_bit_image
node_node_ecs_recipes -->|"configures"| node_node_fluent_bit_image
node_node_windows_recipes -->|"configures"| node_node_windows_line
node_node_examples -->|"demonstrates"| node_node_fluent_bit_image
node_node_init_entrypoint -->|"implements"| node_node_init_variant
node_node_integration_tests -->|"verifies"| node_node_fluent_bit_image
node_node_integration_tests -->|"verifies"| node_node_plugin_bundle
node_node_load_tests -->|"stresses"| node_node_fluent_bit_image
node_node_local_tests -->|"checks"| node_node_configs
node_node_troubleshooting -->|"diagnoses"| node_node_fluent_bit_image

click node_node_linux_line "https://github.com/aws/aws-for-fluent-bit/blob/mainline/linux.version"
click node_node_windows_line "https://github.com/aws/aws-for-fluent-bit/blob/mainline/windows.versions"
click node_node_init_variant "https://github.com/aws/aws-for-fluent-bit/tree/mainline/init"
click node_node_runtime_common "https://github.com/aws/aws-for-fluent-bit/tree/mainline/scripts/dockerfiles/runtime"
click node_node_build_common "https://github.com/aws/aws-for-fluent-bit/tree/mainline/scripts/dockerfiles/build"
click node_node_make_scripts "https://github.com/aws/aws-for-fluent-bit/tree/mainline/Makefile"
click node_node_windows_build "https://github.com/aws/aws-for-fluent-bit/blob/mainline/scripts/build_windows_image.ps1"
click node_node_release_flow "https://github.com/aws/aws-for-fluent-bit/blob/mainline/.github/workflows/initiaterelease.yml"
click node_node_plugin_build "https://github.com/aws/aws-for-fluent-bit/blob/mainline/scripts/build_plugins.sh"
click node_node_configs "https://github.com/aws/aws-for-fluent-bit/blob/mainline/fluent-bit.conf"
click node_node_ecs_recipes "https://github.com/aws/aws-for-fluent-bit/tree/mainline/ecs"
click node_node_windows_recipes "https://github.com/aws/aws-for-fluent-bit/tree/mainline/ecs_windows_forward_daemon"
click node_node_examples "https://github.com/aws/aws-for-fluent-bit/tree/mainline/examples/fluent-bit"
click node_node_init_entrypoint "https://github.com/aws/aws-for-fluent-bit/blob/mainline/init/fluent_bit_init_process.go"
click node_node_integration_tests "https://github.com/aws/aws-for-fluent-bit/tree/mainline/integ"
click node_node_load_tests "https://github.com/aws/aws-for-fluent-bit/tree/mainline/load_tests"
click node_node_local_tests "https://github.com/aws/aws-for-fluent-bit/tree/mainline/tests"
click node_node_troubleshooting "https://github.com/aws/aws-for-fluent-bit/tree/mainline/troubleshooting"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_fluent_bit_image,node_node_linux_line,node_node_windows_line,node_node_init_variant,node_node_runtime_common,node_node_init_entrypoint toneBlue
class node_node_plugin_bundle,node_node_plugin_build toneAmber
class node_node_build_common,node_node_make_scripts,node_node_windows_build,node_node_release_flow toneMint
class node_node_configs,node_node_ecs_recipes,node_node_windows_recipes,node_node_examples toneRose
class node_node_integration_tests,node_node_load_tests,node_node_local_tests toneIndigo
class node_node_troubleshooting toneTeal
```

## Components

### Image Runtime

Fluent Bit container variants and startup layers

- **Fluent Bit Image**: container runtime · Versioned Linux and Windows images that bundle Fluent Bit with AWS extensions
- [Linux Line](https://github.com/aws/aws-for-fluent-bit/blob/mainline/linux.version): image track · Amazon Linux based amd64/arm64 release path · `linux.version`
- [Windows Line](https://github.com/aws/aws-for-fluent-bit/blob/mainline/windows.versions): image track · Server Core based release path with separate tagging and monthly rebuilds · `windows.versions`
- [Init Variant](https://github.com/aws/aws-for-fluent-bit/tree/mainline/init): startup layer · Wrapper process used by init image tags for multi-config startup · `init`
- [Runtime Dockerfiles](https://github.com/aws/aws-for-fluent-bit/tree/mainline/scripts/dockerfiles/runtime): image assembly · Common, debug, valgrind, and init runtime image definitions · `scripts/dockerfiles/runtime`
- [Init Process](https://github.com/aws/aws-for-fluent-bit/blob/mainline/init/fluent_bit_init_process.go): go entrypoint · Init wrapper binary and shell entrypoint that prepare Fluent Bit before handoff · `init/fluent_bit_init_process.go`

### AWS Plugins

Bundled output plugins compiled into images

- **AWS Plugins**: plugin subsystem · CloudWatch Logs, Kinesis Streams, and Firehose output plugins bundled into images
- [Plugin Build](https://github.com/aws/aws-for-fluent-bit/blob/mainline/scripts/build_plugins.sh): plugin compile · Plugin-only build path and optional external plugin source swapping · `scripts/build_plugins.sh`

### Build & Release

Scripts, Dockerfiles, and publishing flows

- [Build Dockerfiles](https://github.com/aws/aws-for-fluent-bit/tree/mainline/scripts/dockerfiles/build): compile stages · Shared build, dependency, parser, and compile stages for image assembly · `scripts/dockerfiles/build`
- [Build Scripts](https://github.com/aws/aws-for-fluent-bit/tree/mainline/Makefile): orchestration · Makefile and shell scripts that drive local and release builds · `Makefile`
- [Windows Builders](https://github.com/aws/aws-for-fluent-bit/blob/mainline/scripts/build_windows_image.ps1): orchestration · PowerShell and Windows Dockerfile paths for building and packaging Windows images · `scripts/build_windows_image.ps1`
- [Release Flow](https://github.com/aws/aws-for-fluent-bit/blob/mainline/.github/workflows/initiaterelease.yml): publishing · Workflows and buildspecs for initiation, publishing, sync, and manifest creation · `.github/workflows/initiaterelease.yml`

### Configs & Examples

Reference configurations and deployment recipes

- [Core Configs](https://github.com/aws/aws-for-fluent-bit/blob/mainline/fluent-bit.conf): fluent-bit configs · Root configs and parser/normalization rules used by the runtime image · `fluent-bit.conf`
- [ECS Recipes](https://github.com/aws/aws-for-fluent-bit/tree/mainline/ecs): deployment configs · Linux ECS and FireLens-style configs for AWS destinations · `ecs`
- [Windows Recipes](https://github.com/aws/aws-for-fluent-bit/tree/mainline/ecs_windows_forward_daemon): deployment configs · Windows ECS daemon-style forwarding configs and curated deployment files · `ecs_windows_forward_daemon`
- [Examples](https://github.com/aws/aws-for-fluent-bit/tree/mainline/examples/fluent-bit): sample configs · Example Fluent Bit and Systems Manager deployment recipes · `examples/fluent-bit`

### Verification

Integration, load, and local test layers

- [Integration Tests](https://github.com/aws/aws-for-fluent-bit/tree/mainline/integ): end-to-end · Short checks against real AWS services and containerized validators · `integ`
- [Load Tests](https://github.com/aws/aws-for-fluent-bit/tree/mainline/load_tests): throughput tests · Resource provisioning and stability checks for log-loss and throughput · `load_tests`
- [Local Tests](https://github.com/aws/aws-for-fluent-bit/tree/mainline/tests): fast tests · Compose-based, Lua filter, and local validator tests · `tests`

### Support

Docs and troubleshooting assets

- [Troubleshooting](https://github.com/aws/aws-for-fluent-bit/tree/mainline/troubleshooting): diagnostics · Debugging, profiling, VS Code, and repro tooling for image and runtime issues · `troubleshooting`

## Connections

- Fluent Bit Image → Linux Line: packages · Linux release track defines the base for Linux images
- Fluent Bit Image → Windows Line: packages · Windows release track defines the base for Windows images
- Fluent Bit Image → Init Variant: includes · Init image tags add a startup wrapper before Fluent Bit
- Linux Line → Runtime Dockerfiles: builds from · Linux images are assembled from runtime Dockerfiles
- Windows Line → Windows Builders: builds from · Windows images use PowerShell and Windows-specific build paths
- Runtime Dockerfiles → Fluent Bit Image: assembles · Runtime Dockerfiles produce the final container image variants
- Build Dockerfiles → Runtime Dockerfiles: feeds · Shared build stages compile binaries and seed runtime layers
- Build Scripts → Build Dockerfiles: drives · Make and shell scripts orchestrate common build stages
- Build Scripts → Plugin Build: drives · Build scripts can compile plugin-only artifacts
- Windows Builders → Release Flow: publishes · Windows build scripts feed publish and sync workflows
- Release Flow → Fluent Bit Image: publishes · Release workflows push tagged image artifacts to registries
- Plugin Build → AWS Plugins: produces · Plugin build path produces the bundled AWS output plugins
- AWS Plugins → Fluent Bit Image: embedded in · Plugins are compiled into the final runtime images
- Core Configs → Fluent Bit Image: configures · Root configs shape the default runtime behavior
- ECS Recipes → Fluent Bit Image: configures · ECS recipes demonstrate production Linux log pipelines
- Windows Recipes → Windows Line: configures · Windows recipes target the Windows image line
- Examples → Fluent Bit Image: demonstrates · Examples show alternative Fluent Bit deployment patterns
- Init Process → Init Variant: implements · Go binary and shell entrypoint implement the init boundary
- Integration Tests → Fluent Bit Image: verifies · Integration tests exercise image behavior against AWS services
- Integration Tests → AWS Plugins: verifies · End-to-end checks validate plugin outputs and delivery paths
- Load Tests → Fluent Bit Image: stresses · Load tests measure throughput and log-loss on image builds
- Local Tests → Core Configs: checks · Local tests validate config and filter behavior
- Troubleshooting → Fluent Bit Image: diagnoses · Debug and profiling assets support image and runtime investigation
