How fluent/fluent-plugin-s3 works
Project type and purpose
• This is an open-source Fluentd plugin (a RubyGem) that provides both an “output” plugin (buffers event logs locally, compresses them, and uploads to Amazon S3) and an “input” plugin (polls SQS notifications, reads objects from S3, and feeds them back into Fluentd).
• Key use case: reliable, time-based log partitioning/buffering and transfer between Fluentd and S3 (with optional compression algorithms).
File structure
• Top‐level artifacts:
• fluent-plugin-s3.gemspec, Gemfile, Rakefile, VERSION → RubyGem packaging, build, CI
• lib/fluent/plugin/in_s3.rb → input plugin entrypoint
• lib/fluent/plugin/out_s3.rb → output plugin entrypoint
• lib/fluent/plugin/s3_compressor_*.rb & s3_extractor_*.rb → modular compression/decompression strategies
• docs/ → configuration, how-to, migration guides
• test/ → unit tests for in_s3 and out_s3
• CI & tooling: .github/workflows, appveyor.yml, Travis badge in README.
README for additional insights
• The README confirms the buffering logic uses event timestamps (not arrival times) to split files.
• The input plugin uses an SQS queue (must be pre-configured with S3 event notifications).
• Supported Ruby versions and Fluentd versions.
Components
a. Main components:
• Fluentd core (event pipeline)
• Output plugin (out_s3):
• Local file buffer
• Compressor modules (gzip, lzma2, lzo, parquet, zstd)
• S3 uploader
• Input plugin (in_s3):
• SQS poller
• S3 object fetcher
• Extractor modules (gzip, lzma2, lzo)
• AWS services: S3 bucket, SQS queue
• Local filesystem (for buffer storage)
b. Relationships & interactions:
• Fluentd → out_s3 plugin → write to local buffer segmented by timestamp
• out_s3 compressor → generate compressed blobs → invoke AWS SDK to PUT to S3
• S3 event triggers SQS notification → in_s3 polls SQS → obtains object key → GET from S3
• in_s3 extractor → decompress payload → emit events back into Fluentd pipeline
c. Architectural patterns:
• Plugin‐based extension of Fluentd core
• Strategy pattern for compressors/extractors (modular .rb files)
• Buffering and batching for throughput and fault tolerance
• Event‐driven integration with AWS via S3 event notifications → SQS queue
d. Key technologies:
• Ruby 2.x, Fluentd v0.14+ or v0.12
• AWS SDK for Ruby (S3, SQS)
• External compression commands (for some compressors)
• Rake/Gem for build, .gemspec for packaging