How aws/aws-for-fluent-bit works
1. What this repository is
- This is the build and release repo for the AWS for Fluent Bit container images, not the upstream Fluent Bit runtime itself.
- Its main output is versioned Linux and Windows container images that bundle Fluent Bit plus AWS-specific output plugins and supporting tooling.
2. Primary runtime boundary
- The core runtime is Fluent Bit running inside a container image.
- The image layers in AWS plugins, default configs, and in some variants an init process for multi-config startup.
- The repo’s architecture is therefore centered on producing immutable image artifacts rather than a long-running application service.
3. Bundled AWS plugin subsystem
- The key functional extension is the set of AWS output plugins: CloudWatch Logs, Kinesis Streams, and Kinesis Firehose.
- These are treated as separately developed components that are compiled and packaged into the final image.
- The repo supports swapping in forked plugin sources for development, which shows the plugins are a distinct subsystem from the image assembly pipeline.
4. Image build and packaging pipeline
- Makefile, scripts/, and scripts/dockerfiles/ define the build graph for release, dev, debug, init, and plugin-only images.
- Dockerfiles are split by purpose: common build stages, runtime stages, debug/valgrind variants, and Windows-specific images.
- Build inputs come from linux.version, windows.versions, and the version marker files in the repo root, which control what gets assembled into each image tag.
5. Linux vs Windows product lines
- Linux images and Windows images are handled as separate release tracks with different rebuild policies and tagging rules.
- Linux uses Amazon Linux base images and supports amd64/arm64 multi-arch tags.
- Windows uses Server Core base images, publishes separate Windows tags, and rebuilds monthly to absorb Microsoft security updates.
6. Init-process variant
- The init/ subtree implements an init process used by “init” image tags.
- This variant exists to support multi-config startup behavior, especially for FireLens-style deployments where a wrapper process prepares Fluent Bit before handing off to it.
- Architecturally, this is a small but important startup boundary between the container entrypoint and Fluent Bit itself.
7. Configuration and example topology
- Root-level configs, ecs/, ecs_windows_forward_daemon/, examples/, use_cases/, and tests/ contain deployment recipes rather than application logic.
- These configs show the intended data flow: application logs enter Fluent Bit via forward, tail, stdout, or socket-style inputs; filters transform records; outputs ship them to AWS destinations.
- The filters/replace_dots.lua script and parser configs indicate that log enrichment and normalization are first-class concerns.
8. AWS destination pipelines
- The main integration targets are CloudWatch Logs, Kinesis, Firehose, and S3.
- The repo’s sample and ECS/Windows configs are organized around these destinations, so the architecture is effectively “input → filter → AWS output.”
- S3 is also used as a validation and cleanup target in testing and load-test workflows, not only as a production sink.
9. Testing hierarchy
- The repo has three major test layers: unit/small tests, integration tests, and load/stability tests.
- integ/ runs short end-to-end checks against real AWS resources or containerized validators.
- load_tests/ creates test infrastructure and validates throughput/log-loss thresholds.
- tests/ contains local compose-based and Lua filter tests for faster feedback.
10. Test infrastructure provisioning
- The repo provisions AWS resources through CDK-style app folders under integ/test_init/ and load_tests/create_testing_resources/.
- These helpers create the external environment the image needs to prove correctness against real AWS services.
- This means the system boundary is not just container runtime vs source code; it also includes cloud resources required for verification.
11. Release and publishing pipeline
- The GitHub Actions workflows and buildspec*.yml files drive release initiation, title linting, image publishing, and sync jobs.
- Publishing targets include Docker Hub, Amazon ECR, public ECR, and SSM public parameters.
- The repo’s release process is tightly coupled to image tagging and version propagation across registries and parameter stores.
12. Operational support and diagnostics
- docs/, troubleshooting/, and debug/valgrind Dockerfiles are part of the product architecture because they shape how the image is operated and investigated.
- Debug variants exist specifically for crash analysis and memory tooling, which indicates a deliberate split between production runtime images and diagnostic images.
13. Windows deployment support
- ecs_windows_forward_daemon/ and the Windows-specific integration/config folders provide curated configs for ECS daemon-style forwarding on Windows.
- This is a separate deployment pattern from Linux FireLens-style usage and reflects a distinct runtime environment and networking model.
14. Overall system picture
- At a high level, the repo builds a family of container images that convert application logs into AWS logging/streaming/storage services.
- The major architectural boundaries are: upstream Fluent Bit source, AWS plugin code, image assembly/runtime layers, cloud test infrastructure, and release/publishing infrastructure.
Main entry points: windows.versions Windows Line · runtime Runtime Dockerfiles · Makefile Build Scripts · integ Integration Tests
Components 20
Image Runtime
Fluent Bit container variants and startup layers
- Fluent Bit Image · Versioned Linux and Windows images that bundle Fluent Bit with AWS extensions
- Linux Line · Amazon Linux based amd64/arm64 release path
linux.version - Windows Line · Server Core based release path with separate tagging and monthly rebuilds
windows.versions - Init Variant · Wrapper process used by init image tags for multi-config startup
init - Runtime Dockerfiles · Common, debug, valgrind, and init runtime image definitions
scripts/dockerfiles/runtime - Init Process · Init wrapper binary and shell entrypoint that prepare Fluent Bit before handoff
init/fluent_bit_init_process.go
AWS Plugins
Bundled output plugins compiled into images
- AWS Plugins · CloudWatch Logs, Kinesis Streams, and Firehose output plugins bundled into images
- Plugin Build · Plugin-only build path and optional external plugin source swapping
scripts/build_plugins.sh
Build & Release
Scripts, Dockerfiles, and publishing flows
- Build Dockerfiles · Shared build, dependency, parser, and compile stages for image assembly
scripts/dockerfiles/build - Build Scripts · Makefile and shell scripts that drive local and release builds
Makefile - Windows Builders · PowerShell and Windows Dockerfile paths for building and packaging Windows images
scripts/build_windows_image.ps1 - Release Flow · Workflows and buildspecs for initiation, publishing, sync, and manifest creation
.github/workflows/initiaterelease.yml
Configs & Examples
Reference configurations and deployment recipes
- Core Configs · Root configs and parser/normalization rules used by the runtime image
fluent-bit.conf - ECS Recipes · Linux ECS and FireLens-style configs for AWS destinations
ecs - Windows Recipes · Windows ECS daemon-style forwarding configs and curated deployment files
ecs_windows_forward_daemon - Examples · Example Fluent Bit and Systems Manager deployment recipes
examples/fluent-bit
Verification
Integration, load, and local test layers
- Integration Tests · Short checks against real AWS services and containerized validators
integ - Load Tests · Resource provisioning and stability checks for log-loss and throughput
load_tests - Local Tests · Compose-based, Lua filter, and local validator tests
tests
Support
Docs and troubleshooting assets
- Troubleshooting · Debugging, profiling, VS Code, and repro tooling for image and runtime issues
troubleshooting
Connections 23
- Fluent Bit Image → Linux Line: packages · Linux release track defines the base for Linux images
- Fluent Bit Image → Windows Line: packages · Windows release track defines the base for Windows images
- Fluent Bit Image → Init Variant: includes · Init image tags add a startup wrapper before Fluent Bit
- Linux Line → Runtime Dockerfiles: builds from · Linux images are assembled from runtime Dockerfiles
- Windows Line → Windows Builders: builds from · Windows images use PowerShell and Windows-specific build paths
- Runtime Dockerfiles → Fluent Bit Image: assembles · Runtime Dockerfiles produce the final container image variants
- Build Dockerfiles → Runtime Dockerfiles: feeds · Shared build stages compile binaries and seed runtime layers
- Build Scripts → Build Dockerfiles: drives · Make and shell scripts orchestrate common build stages
- Build Scripts → Plugin Build: drives · Build scripts can compile plugin-only artifacts
- Windows Builders → Release Flow: publishes · Windows build scripts feed publish and sync workflows
- Release Flow → Fluent Bit Image: publishes · Release workflows push tagged image artifacts to registries
- Plugin Build → AWS Plugins: produces · Plugin build path produces the bundled AWS output plugins
- AWS Plugins → Fluent Bit Image: embedded in · Plugins are compiled into the final runtime images
- Core Configs → Fluent Bit Image: configures · Root configs shape the default runtime behavior
- ECS Recipes → Fluent Bit Image: configures · ECS recipes demonstrate production Linux log pipelines
- Windows Recipes → Windows Line: configures · Windows recipes target the Windows image line
- Examples → Fluent Bit Image: demonstrates · Examples show alternative Fluent Bit deployment patterns
- Init Process → Init Variant: implements · Go binary and shell entrypoint implement the init boundary
- Integration Tests → Fluent Bit Image: verifies · Integration tests exercise image behavior against AWS services
- Integration Tests → AWS Plugins: verifies · End-to-end checks validate plugin outputs and delivery paths
- Load Tests → Fluent Bit Image: stresses · Load tests measure throughput and log-loss on image builds
- Local Tests → Core Configs: checks · Local tests validate config and filter behavior
- Troubleshooting → Fluent Bit Image: diagnoses · Debug and profiling assets support image and runtime investigation