How aiven-open/http-connector-for-apache-kafka works
Project Type and Purpose
• This is a Java-based Apache Kafka Connect sink connector plugin that delivers Kafka records over HTTP/S.
• It plugs into a Kafka Connect runtime, reads from Kafka topics, converts and batches records, and posts them via HTTP(S) to arbitrary endpoints.
File Structure Highlights
• src/main/java/io/aiven/kafka/connect/http
• HttpSinkConnector (the Connector class)
• HttpSinkTask (the SinkTask implementation)
• config/* (all configuration definitions and validators)
• converter/* (record-to-payload converters)
• recordsender/* (batch vs single record senders)
• sender/* (HTTP client implementations, factories, OAuth2 support, SSL trust store)
• src/integration-test
• Mock HTTP server, integration test runners against Docker containers
• build.gradle.kts, gradlew, gradle/
• Gradle build scripts, plugin publishing, Checkstyle and Spotless config
How
a. Main Components
• Kafka Cluster (source of records)
• Kafka Connect Worker
• Connector Plugin (HttpSinkConnector)
• SinkTask(s) (HttpSinkTask instances)
• Converter Module (JsonRecordValueConverter, etc.)
• RecordSender Module (BatchRecordSender or SingleRecordSender)
• HttpSender Module (DefaultHttpSender, OAuth2HttpSender, etc.)
• Configuration Module (HttpSinkConfig + validators)
• External Services
• HTTP Endpoint(s) (your downstream service)
• (Optional) OAuth2 Authorization Server
b. Relationships and Data Flow
1. Kafka Connect Worker loads HttpSinkConnector plugin
2. Connector spawns one or more HttpSinkTask instances
3. Each SinkTask polls Kafka Cluster for new records
4. Records pass through Converter → produce JSON/other payloads
5. Payloads handed to RecordSender (batching or single)
6. RecordSender uses an HttpSender (built via HttpSenderFactory)
• May first fetch OAuth2 token via OAuth2AccessTokenHttpSender
7. HttpSender constructs requests via HttpRequestBuilder, applies headers, SSL, auth
8. HTTP POST(s) sent to External HTTP Endpoint
9. HttpResponseHandler processes HTTP responses, triggers retry logic if enabled
c. Key Architectural Patterns
• Plugin Architecture: implements Kafka Connect’s Connector and SinkTask interfaces
• Factory Pattern: HttpSenderFactory to instantiate correct HttpSender subtype
• Strategy Pattern: different RecordSender (batch vs single), Converter, AuthorizationType
• Builder Pattern: HttpRequestBuilder to assemble HTTP requests flexibly
• Layered Modules: clear separation between config, conversion, sending, networking
d. Technologies & Frameworks
• Java 11+
• Apache Kafka Connect API
• Gradle (build, integrationTest, code style checks)
• HTTP client (under the hood, e.g., Java HttpClient or Apache HTTPComponents)
• OAuth2 support (custom HTTP flows)
Detail Level
• Retry logic and OAuth2 token fetch as separate interactions if relevant.
• Unit vs integration test mock server in a side-note for QA architecture.