# redpanda-data/connect architecture

> Architecture diagram of the GitHub repository redpanda-data/connect, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/redpanda-data/connect
- Repository: https://github.com/redpanda-data/connect

## Overview

##### Project type and purpose
- This is a full-stack, high-performance Go stream-processing tool (Redpanda Connect) shipped as a single binary or Docker image.
- Purpose: connect various sources (Kafka, AWS, GCP, HTTP, databases, etc.) to sinks, apply user-defined transformations (Bloblang), and expose health/metrics/tracing endpoints.

##### File structure
- Top-level “cmd/…” directories: multiple entry points (redpanda-connect, redpanda-connect-cloud, ‑ai, ‑community, plus serverless handlers).
- “internal/cli”: command-line flag parsing, subcommands.
- “internal/impl”: protocol-specific input/output/processor implementations (one subpackage per connector).
- “internal/asyncroutine”, “internal/dispatch”: core scheduler, batching, routing.
- “internal/agent” and “internal/mcp”: AI-agent and micro-control-plane support.
- “public”: plugin API bundles for custom Go plugins.
- “config” & “resources”: sample YAML configs, Docker Compose, Dockerfiles.
- “proto”: gRPC definitions for runtime and control APIs.
- “internal/telemetry” & “internal/tracing”: metrics and OpenTelemetry integration.
- Health & metrics served over HTTP in “internal/serverless” & core engine.

##### Key README insights
- Single YAML config declares `input`, `pipeline.processors`, and `output`.
- Delivery guarantees: in-process transactions, at-least-once.
- Exposes HTTP endpoints `/ping` and `/ready`.
- Metrics: Statsd, Prometheus, JSON HTTP.
- Tracing: OpenTelemetry events per processor.

##### Main components
a. **CLI & Bootstrap**
- Entrypoint (e.g. `rpk connect run`)
- Config loader & validator
b. **Core Engine**
- Dispatcher/async-routine manager (batchers, workers)
- Bloblang interpreter for transformations
c. **Connector Adapters**
- Input adapters (e.g. Kafka, AWS SQS, HTTP Server)
- Output adapters (e.g. Redis Streams, Kafka, GCP Pub/Sub)
- Processor adapters (built-in processors, 3rd-party plugins)
d. **Agent & MCP Subsystems (optional)**
- AI agent runtime (gRPC agent services)
- MCP (micro control plane) server for remote management
e. **Observability**
- HTTP endpoints: `/ping`, `/ready`
- Metrics collector & exporter (Prometheus, Statsd, HTTP JSON)
- Tracing exporter (OTLP, Jaeger)
f. **External Systems**
- Brokers & Datastores: Kafka/Redpanda, AWS/GCP services, Databases, Caches, File Stores, etc.
- Plugin registry (public API modules, Docker image)

##### Relationships & interactions
- The CLI initializes the Core Engine, injects the loaded config.
- Core Engine spins up an Input adapter → pushes messages into dispatcher → workers execute Bloblang/processors → dispatch to Output adapter.
- Health checks read adapter states; metrics/traces are emitted throughout.
- Agent/MCP connect to gRPC endpoints inside the Engine.
- Plugin API modules (in “public”) are loaded at startup or via build tags.

##### Architectural patterns & design principles
- **Modularity**: each connector in its own package under `internal/impl`.
- **Plug-in architecture**: compile-time plugins via build tags; runtime hooks via `public` package.
- **Async pipeline**: non-blocking batching and back-pressure (`internal/asyncroutine`).
- **Configuration-driven**: declare pipelines in YAML, minimal code changes.
- **Sidecar observability**: HTTP for liveness/readiness, metrics, tracing.

##### Relevant technologies & frameworks
- Go modules for dependency management (`go.mod`)
- gRPC / protobuf for agent & runtime APIs (`proto/…`)
- Docker multi-stage builds (resources/docker/Dockerfile)
- OpenTelemetry, Prometheus client, Statsd library
- Build system: Makefile, goreleaser, golangci-lint

##### Detail
- Don’t omit the agent/MCP pieces if they’re relevant to the chosen binary (AI vs Cloud vs Community).
- Plugin API surface under “public/…” to show extensibility points.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Class definitions
    classDef core fill:#D0E8FF,stroke:#0366d6,color:#0366d6;
    classDef connector fill:#DFF4DD,stroke:#22863A,color:#22863A;
    classDef observability fill:#FFEFD5,stroke:#D66F00,color:#D66F00;
    classDef external fill:#F0F0F0,stroke:#6A737D,color:#6A737D;

    %% Core Binary Layer
    subgraph "Core Binary" 
        direction TB
        CLI["CLI & Bootstrap"]:::core
        Config["Config Loader & Validator"]:::core
        Engine["Core Engine"]:::core

        subgraph "Engine Components"
            direction TB
            Dispatch["Dispatcher & Async Manager"]:::core
            Async["Async Routines (Batcher & Workers)"]:::core
            Bloblang["Bloblang Interpreter"]:::core
        end

        subgraph "Subsystems"
            direction TB
            Agent["AI Agent Subsystem"]:::core
            MCP["Micro Control Plane (MCP)"]:::core
        end

        subgraph "Plugin API Surface"
            direction TB
            PublicAPI["Plugin API (public)"]:::core
        end

        subgraph "Observability"
            direction TB
            HTTP["HTTP Endpoints (/ping, /ready)"]:::observability
            Metrics["Metrics Collector & Exporter"]:::observability
            Tracing["Tracing Exporter"]:::observability
        end
    end

    %% Connectors
    subgraph "Connector Adapters" 
        direction TB
        InputKafka["Input: Kafka Adapter"]:::connector
        InputAWS["Input: AWS SQS/Kinesis Adapter"]:::connector
        OutputRedis["Output: Redis Streams Adapter"]:::connector
        OutputGCP["Output: GCP Pub/Sub & BigQuery Adapter"]:::connector
        ProcBuiltin["Built-in Processor Adapters"]:::connector
        ProcPlugin["3rd-Party Processor Plugins"]:::connector
    end

    %% External Systems
    subgraph "External Systems"
        direction TB
        Brokers["Kafka/Redpanda Brokers"]:::external
        Cloud["AWS/GCP Services"]:::external
        Datastore["Databases & Caches"]:::external
        Files["File Stores"]:::external
        Monitoring["Prometheus/Statsd/Jaeger"]:::external
    end

    %% Connections
    CLI --> Config
    Config --> Engine
    Engine --> Dispatch
    Dispatch --> Async
    Async --> Bloblang
    Bloblang --> InputKafka
    Bloblang --> InputAWS
    Bloblang --> ProcBuiltin
    Bloblang --> ProcPlugin
    ProcBuiltin --> OutputRedis
    ProcPlugin --> OutputGCP
    OutputRedis --> Brokers
    OutputGCP --> Cloud
    InputKafka --> Brokers
    InputAWS --> Cloud

    Engine --> HTTP
    Engine --> Metrics
    Engine --> Tracing

    Engine --> Agent
    Engine --> MCP

    Metrics --> Monitoring
    Tracing --> Monitoring
    HTTP --> Monitoring

    PublicAPI --> Engine

    %% Click Events
    click CLI "https://github.com/redpanda-data/connect/blob/main/cmd/redpanda-connect/main.go"
    click Config "https://github.com/redpanda-data/connect/blob/main/internal/cli/flags_common.go"
    click Dispatch "https://github.com/redpanda-data/connect/blob/main/internal/dispatch/detect.go"
    click Async "https://github.com/redpanda-data/connect/blob/main/internal/asyncroutine/batcher.go"
    click Bloblang "https://github.com/redpanda-data/connect/blob/main/internal/impl/lang/bloblang.go"
    click InputKafka "https://github.com/redpanda-data/connect/blob/main/internal/impl/kafka/input_redpanda.go"
    click InputAWS "https://github.com/redpanda-data/connect/blob/main/internal/impl/aws/input_sqs.go"
    click OutputRedis "https://github.com/redpanda-data/connect/blob/main/internal/impl/redis/output_streams.go"
    click OutputGCP "https://github.com/redpanda-data/connect/blob/main/internal/impl/gcp/output_pubsub.go"
    click ProcBuiltin "https://github.com/redpanda-data/connect/blob/main/internal/impl/awk/processor.go"
    click ProcPlugin "https://github.com/redpanda-data/connect/tree/main/public/components"
    click Agent "https://github.com/redpanda-data/connect/blob/main/internal/agent/agent.go"
    click MCP "https://github.com/redpanda-data/connect/blob/main/internal/mcp/mcp.go"
    click HTTP "https://github.com/redpanda-data/connect/blob/main/internal/serverless/handler.go"
    click Metrics "https://github.com/redpanda-data/connect/blob/main/internal/telemetry/telemetry.go"
    click Tracing "https://github.com/redpanda-data/connect/blob/main/internal/tracing/custom_ids.go"
    click PublicAPI "https://github.com/redpanda-data/connect/blob/main/public/bundle/enterprise/package.go"

    %% Styles
    class CLI,Config,Engine,Dispatch,Async,Bloblang,Agent,MCP,PublicAPI core
    class InputKafka,InputAWS,OutputRedis,OutputGCP,ProcBuiltin,ProcPlugin connector
    class HTTP,Metrics,Tracing observability
    class Brokers,Cloud,Datastore,Files,Monitoring external
```
