# aiven-open/http-connector-for-apache-kafka architecture

> Architecture diagram of the GitHub repository aiven-open/http-connector-for-apache-kafka, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aiven-open/http-connector-for-apache-kafka
- Repository: https://github.com/aiven-open/http-connector-for-apache-kafka

## Overview

##### Project Type and Purpose
- This is a Java-based Apache Kafka Connect **sink connector** plugin that delivers Kafka records over HTTP/S.
- It plugs into a Kafka Connect runtime, reads from Kafka topics, converts and batches records, and posts them via HTTP(S) to arbitrary endpoints.

##### File Structure Highlights
- **src/main/java/io/aiven/kafka/connect/http**
- HttpSinkConnector (the Connector class)
- HttpSinkTask (the SinkTask implementation)
- config/* (all configuration definitions and validators)
- converter/* (record-to-payload converters)
- recordsender/* (batch vs single record senders)
- sender/* (HTTP client implementations, factories, OAuth2 support, SSL trust store)
- **src/integration-test**
- Mock HTTP server, integration test runners against Docker containers
- **build.gradle.kts, gradlew, gradle/**
- Gradle build scripts, plugin publishing, Checkstyle and Spotless config

##### How
a. **Main Components**
- **Kafka Cluster** (source of records)
- **Kafka Connect Worker**
- Connector Plugin (HttpSinkConnector)
- SinkTask(s) (HttpSinkTask instances)
- Converter Module (JsonRecordValueConverter, etc.)
- RecordSender Module (BatchRecordSender or SingleRecordSender)
- HttpSender Module (DefaultHttpSender, OAuth2HttpSender, etc.)
- Configuration Module (HttpSinkConfig + validators)
- **External Services**
- HTTP Endpoint(s) (your downstream service)
- (Optional) OAuth2 Authorization Server

b. **Relationships and Data Flow**
1. Kafka Connect Worker loads **HttpSinkConnector** plugin
2. Connector spawns one or more **HttpSinkTask** instances
3. Each SinkTask polls **Kafka Cluster** for new records
4. Records pass through **Converter** → produce JSON/other payloads
5. Payloads handed to **RecordSender** (batching or single)
6. RecordSender uses an **HttpSender** (built via HttpSenderFactory)
- May first fetch OAuth2 token via OAuth2AccessTokenHttpSender
7. HttpSender constructs requests via HttpRequestBuilder, applies headers, SSL, auth
8. HTTP POST(s) sent to **External HTTP Endpoint**
9. HttpResponseHandler processes HTTP responses, triggers retry logic if enabled

c. **Key Architectural Patterns**
- **Plugin Architecture:** implements Kafka Connect’s Connector and SinkTask interfaces
- **Factory Pattern:** HttpSenderFactory to instantiate correct HttpSender subtype
- **Strategy Pattern:** different RecordSender (batch vs single), Converter, AuthorizationType
- **Builder Pattern:** HttpRequestBuilder to assemble HTTP requests flexibly
- **Layered Modules:** clear separation between config, conversion, sending, networking

d. **Technologies & Frameworks**
- Java 11+
- Apache Kafka Connect API
- Gradle (build, integrationTest, code style checks)
- HTTP client (under the hood, e.g., Java HttpClient or Apache HTTPComponents)
- OAuth2 support (custom HTTP flows)

##### Detail Level
- Retry logic and OAuth2 token fetch as separate interactions if relevant.
- Unit vs integration test mock server in a side-note for QA architecture.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Systems
    KafkaCluster["Kafka Cluster"]:::external
    OAuth2Server["OAuth2 Authorization Server"]:::external
    HTTPEndpoint["External HTTP Endpoint"]:::external

    %% Kafka Connect Worker
    subgraph "Kafka Connect Worker"
        direction TB
        HttpSinkConnector["HttpSinkConnector"]:::core
        subgraph "Sink Task Lifecycle"
            direction TB
            HttpSinkTask["HttpSinkTask"]:::core
            subgraph "HttpSinkTask Process"
                direction TB
                subgraph "Configuration Module"
                    direction TB
                    HttpSinkConfig["HttpSinkConfig"]:::module
                    AuthorizationType["AuthorizationType"]:::module
                    OAuth2AuthorizationMode["OAuth2AuthorizationMode"]:::module
                    FixedSetRecommender["FixedSetRecommender"]:::module
                    UrlValidator["UrlValidator"]:::module
                    KeyValuePairListValidator["KeyValuePairListValidator"]:::module
                    NonBlankStringValidator["NonBlankStringValidator"]:::module
                end
                subgraph "Converter Module"
                    direction TB
                    RecordValueConverter["RecordValueConverter"]:::module
                    JsonRecordValueConverter["JsonRecordValueConverter"]:::module
                end
                subgraph "RecordSender Module"
                    direction TB
                    RecordSender["RecordSender"]:::module
                    BatchRecordSender["BatchRecordSender"]:::module
                    SingleRecordSender["SingleRecordSender"]:::module
                end
                subgraph "HttpSender Module"
                    direction TB
                    HttpSenderFactory["HttpSenderFactory"]:::module
                    HttpSender["HttpSender"]:::module
                    AbstractHttpSender["AbstractHttpSender"]:::module
                    DefaultHttpSender["DefaultHttpSender"]:::module
                    OAuth2HttpSender["OAuth2HttpSender"]:::auth
                    OAuth2AccessTokenHttpSender["OAuth2AccessTokenHttpSender"]:::auth
                    StaticAuthHttpSender["StaticAuthHttpSender"]:::auth
                end
                HttpRequestBuilder["HttpRequestBuilder"]:::http
                HttpResponseHandler["HttpResponseHandler"]:::http
                subgraph "SSL & Truststore"
                    direction TB
                    SslContextBuilder["SslContextBuilder"]:::module
                    TrustStoreLoader["TrustStoreLoader"]:::module
                end
                OAuth2AccessTokenRequestForm["OAuth2AccessTokenRequestForm"]:::auth
            end
        end
    end

    %% Data Flow
    KafkaCluster -->|"SourceRecord"| HttpSinkTask
    HttpSinkTask -->|"validate config"| HttpSinkConfig
    HttpSinkTask -->|"convert record"| JsonRecordValueConverter
    JsonRecordValueConverter -->|"JSON payload"| RecordSender
    RecordSender -->|"batch/single payload"| HttpSenderFactory
    HttpSenderFactory -->|"instantiate sender"| DefaultHttpSender
    HttpSenderFactory -->|"instantiate sender"| OAuth2HttpSender
    OAuth2HttpSender -->|"fetch token"| OAuth2AccessTokenHttpSender
    OAuth2AccessTokenHttpSender -->|"token request"| OAuth2Server
    DefaultHttpSender -->|"build request"| HttpRequestBuilder
    HttpRequestBuilder -->|"SSL context"| SslContextBuilder
    SslContextBuilder --> TrustStoreLoader
    HttpRequestBuilder --> HTTPEndpoint
    HTTPEndpoint -->|"HTTP response"| HttpResponseHandler
    HttpResponseHandler -->|"retry if needed"| HttpRequestBuilder

    %% Connector to Task
    HttpSinkConnector -->|"spawns tasks"| HttpSinkTask

    %% Styling
    classDef external fill:#ffe4b5,stroke:#daa520
    classDef core fill:#add8e6,stroke:#00008b
    classDef module fill:#90ee90,stroke:#006400
    classDef http fill:#ffb6c1,stroke:#c71585
    classDef auth fill:#dda0dd,stroke:#800080

    %% Click Events
    click HttpSinkConnector "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/HttpSinkConnector.java"
    click HttpSinkTask "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/HttpSinkTask.java"
    click HttpSinkConfig "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/config/HttpSinkConfig.java"
    click AuthorizationType "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/config/AuthorizationType.java"
    click OAuth2AuthorizationMode "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/config/OAuth2AuthorizationMode.java"
    click FixedSetRecommender "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/config/FixedSetRecommender.java"
    click UrlValidator "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/config/UrlValidator.java"
    click KeyValuePairListValidator "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/config/KeyValuePairListValidator.java"
    click NonBlankStringValidator "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/config/NonBlankStringValidator.java"
    click RecordValueConverter "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/converter/RecordValueConverter.java"
    click JsonRecordValueConverter "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/converter/JsonRecordValueConverter.java"
    click RecordSender "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/recordsender/RecordSender.java"
    click BatchRecordSender "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/recordsender/BatchRecordSender.java"
    click SingleRecordSender "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/recordsender/SingleRecordSender.java"
    click HttpSender "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/sender/HttpSender.java"
    click AbstractHttpSender "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/sender/AbstractHttpSender.java"
    click DefaultHttpSender "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/sender/DefaultHttpSender.java"
    click OAuth2HttpSender "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/sender/OAuth2HttpSender.java"
    click OAuth2AccessTokenHttpSender "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/sender/OAuth2AccessTokenHttpSender.java"
    click StaticAuthHttpSender "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/sender/StaticAuthHttpSender.java"
    click HttpSenderFactory "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/sender/HttpSenderFactory.java"
    click HttpRequestBuilder "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/sender/HttpRequestBuilder.java"
    click HttpResponseHandler "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/sender/HttpResponseHandler.java"
    click SslContextBuilder "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/sender/SslContextBuilder.java"
    click TrustStoreLoader "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/sender/TrustStoreLoader.java"
    click OAuth2AccessTokenRequestForm "https://github.com/aiven-open/http-connector-for-apache-kafka/blob/main/src/main/java/io/aiven/kafka/connect/http/sender/request/OAuth2AccessTokenRequestForm.java"
```
