# fluent/fluent-plugin-s3 architecture

> Architecture diagram of the GitHub repository fluent/fluent-plugin-s3, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fluent/fluent-plugin-s3
- Repository: https://github.com/fluent/fluent-plugin-s3

## Overview

##### Project type and purpose
- This is an open-source Fluentd plugin (a RubyGem) that provides both an “output” plugin (buffers event logs locally, compresses them, and uploads to Amazon S3) and an “input” plugin (polls SQS notifications, reads objects from S3, and feeds them back into Fluentd).
- Key use case: reliable, time-based log partitioning/buffering and transfer between Fluentd and S3 (with optional compression algorithms).

##### File structure
- Top‐level artifacts:
- fluent-plugin-s3.gemspec, Gemfile, Rakefile, VERSION → RubyGem packaging, build, CI
- lib/fluent/plugin/in_s3.rb → input plugin entrypoint
- lib/fluent/plugin/out_s3.rb → output plugin entrypoint
- lib/fluent/plugin/s3_compressor_*.rb & s3_extractor_*.rb → modular compression/decompression strategies
- docs/ → configuration, how-to, migration guides
- test/ → unit tests for in_s3 and out_s3
- CI & tooling: .github/workflows, appveyor.yml, Travis badge in README.

##### README for additional insights
- The README confirms the buffering logic uses event timestamps (not arrival times) to split files.
- The input plugin uses an SQS queue (must be pre-configured with S3 event notifications).
- Supported Ruby versions and Fluentd versions.

##### Components
a. Main components:
- Fluentd core (event pipeline)
- Output plugin (out_s3):
- Local file buffer
- Compressor modules (gzip, lzma2, lzo, parquet, zstd)
- S3 uploader
- Input plugin (in_s3):
- SQS poller
- S3 object fetcher
- Extractor modules (gzip, lzma2, lzo)
- AWS services: S3 bucket, SQS queue
- Local filesystem (for buffer storage)
b. Relationships & interactions:
- Fluentd → out_s3 plugin → write to local buffer segmented by timestamp
- out_s3 compressor → generate compressed blobs → invoke AWS SDK to PUT to S3
- S3 event triggers SQS notification → in_s3 polls SQS → obtains object key → GET from S3
- in_s3 extractor → decompress payload → emit events back into Fluentd pipeline
c. Architectural patterns:
- Plugin‐based extension of Fluentd core
- Strategy pattern for compressors/extractors (modular .rb files)
- Buffering and batching for throughput and fault tolerance
- Event‐driven integration with AWS via S3 event notifications → SQS queue
d. Key technologies:
- Ruby 2.x, Fluentd v0.14+ or v0.12
- AWS SDK for Ruby (S3, SQS)
- External compression commands (for some compressors)
- Rake/Gem for build, .gemspec for packaging

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Fluentd Process
    subgraph "Fluentd Process"
        direction TB
        FluentdCore["Fluentd Core"]:::fluent
        FluentdCore -->|send events| OutPlugin
        InPlugin -->|emit events| FluentdCore
    end

    %% Output Plugin
    subgraph "Output Plugin (out_s3)" 
        direction TB
        OutPlugin["out_s3 Plugin"]:::fluent
        OutPlugin --> Buffer["Buffer (time-based roll)"]:::storage
        Buffer --> LocalDisk["Local Disk"]:::storage
        Buffer --> CompressorGroup
        subgraph "Compressor Strategies"
            direction TB
            GzipComp["gzip"]:::strategy
            Lzma2Comp["lzma2"]:::strategy
            LzoComp["lzo"]:::strategy
            ParquetComp["parquet"]:::strategy
            ZstdComp["zstd"]:::strategy
        end
        CompressorGroup --> Uploader["S3 Uploader"]:::fluent
        Uploader --> S3Bucket
    end

    %% AWS Layer
    subgraph "AWS Services"
        direction TB
        S3Bucket["AWS S3 Bucket"]:::aws
        SQSQueue["AWS SQS Queue"]:::aws
    end

    S3Bucket -->|event notification| SQSQueue

    %% Input Plugin
    subgraph "Input Plugin (in_s3)"
        direction TB
        InPlugin["in_s3 Plugin"]:::fluent
        InPlugin --> SQSPoller["SQS Poller"]:::fluent
        SQSPoller --> S3Fetcher["S3 Object Fetcher"]:::fluent
        S3Fetcher --> ExtractorGroup
        subgraph "Extractor Strategies"
            direction TB
            GzipExt["gzip"]:::strategy
            Lzma2Ext["lzma2"]:::strategy
            LzoExt["lzo"]:::strategy
        end
        ExtractorGroup --> InPlugin
    end

    SQSQueue -->|poll| SQSPoller

    %% Tooling & Docs
    subgraph "Tooling & Docs"
        direction TB
        Packaging["Packaging & Build"]:::docs
        ConfigDocs["Configuration Guides"]:::docs
        CredDocs["Credential Lookup Docs"]:::docs
        TestOut["Test Suite for out_s3"]:::test
        TestIn["Test Suite for in_s3"]:::test
    end

    %% Connections
    Packaging -.-> OutPlugin
    ConfigDocs -.-> InPlugin
    CredDocs -.-> OutPlugin

    %% Click Events
    click OutPlugin "https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/out_s3.rb"
    click InPlugin "https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/in_s3.rb"
    click GzipComp "https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/s3_compressor_gzip_command.rb"
    click Lzma2Comp "https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/s3_compressor_lzma2.rb"
    click LzoComp "https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/s3_compressor_lzo.rb"
    click ParquetComp "https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/s3_compressor_parquet.rb"
    click ZstdComp "https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/s3_compressor_zstd.rb"
    click GzipExt "https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/s3_extractor_gzip_command.rb"
    click Lzma2Ext "https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/s3_extractor_lzma2.rb"
    click LzoExt "https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/s3_extractor_lzo.rb"
    click Buffer "https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/out_s3.rb"
    click Uploader "https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/out_s3.rb"
    click SQSPoller "https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/in_s3.rb"
    click S3Fetcher "https://github.com/fluent/fluent-plugin-s3/blob/master/lib/fluent/plugin/in_s3.rb"
    click TestOut "https://github.com/fluent/fluent-plugin-s3/blob/master/test/test_out_s3.rb"
    click TestIn "https://github.com/fluent/fluent-plugin-s3/blob/master/test/test_in_s3.rb"
    click CredDocs "https://github.com/fluent/fluent-plugin-s3/blob/master/docs/credentials.md"
    click ConfigDocs "https://github.com/fluent/fluent-plugin-s3/blob/master/docs/input.md"
    click Packaging "https://github.com/fluent/fluent-plugin-s3/blob/master/fluent-plugin-s3.gemspec"

    %% Styles
    classDef fluent fill:#AED6F1,stroke:#3498DB,color:#154360
    classDef aws fill:#ABEBC6,stroke:#27AE60,color:#145A32
    classDef storage fill:#D5D8DC,stroke:#7B7D7D,color:#1C2833
    classDef strategy fill:#FCF3CF,stroke:#F4D03F,color:#7D6608
    classDef docs fill:#FADBD8,stroke:#EC7063,color:#641E16
    classDef test fill:#E8DAEF,stroke:#8E44AD,color:#4A235A
```
