How redpanda-data/connect works
Project type and purpose
• This is a full-stack, high-performance Go stream-processing tool (Redpanda Connect) shipped as a single binary or Docker image.
• Purpose: connect various sources (Kafka, AWS, GCP, HTTP, databases, etc.) to sinks, apply user-defined transformations (Bloblang), and expose health/metrics/tracing endpoints.
File structure
• Top-level “cmd/…” directories: multiple entry points (redpanda-connect, redpanda-connect-cloud, ‑ai, ‑community, plus serverless handlers).
• “internal/cli”: command-line flag parsing, subcommands.
• “internal/impl”: protocol-specific input/output/processor implementations (one subpackage per connector).
• “internal/asyncroutine”, “internal/dispatch”: core scheduler, batching, routing.
• “internal/agent” and “internal/mcp”: AI-agent and micro-control-plane support.
• “public”: plugin API bundles for custom Go plugins.
• “config” & “resources”: sample YAML configs, Docker Compose, Dockerfiles.
• “proto”: gRPC definitions for runtime and control APIs.
• “internal/telemetry” & “internal/tracing”: metrics and OpenTelemetry integration.
• Health & metrics served over HTTP in “internal/serverless” & core engine.
Key README insights
• Single YAML config declares input, pipeline.processors, and output.
• Delivery guarantees: in-process transactions, at-least-once.
• Exposes HTTP endpoints /ping and /ready.
• Metrics: Statsd, Prometheus, JSON HTTP.
• Tracing: OpenTelemetry events per processor.
Main components
a. CLI & Bootstrap
• Entrypoint (e.g. rpk connect run)
• Config loader & validator
b. Core Engine
• Dispatcher/async-routine manager (batchers, workers)
• Bloblang interpreter for transformations
c. Connector Adapters
• Input adapters (e.g. Kafka, AWS SQS, HTTP Server)
• Output adapters (e.g. Redis Streams, Kafka, GCP Pub/Sub)
• Processor adapters (built-in processors, 3rd-party plugins)
d. Agent & MCP Subsystems (optional)
• AI agent runtime (gRPC agent services)
• MCP (micro control plane) server for remote management
e. Observability
• HTTP endpoints: /ping, /ready
• Metrics collector & exporter (Prometheus, Statsd, HTTP JSON)
• Tracing exporter (OTLP, Jaeger)
f. External Systems
• Brokers & Datastores: Kafka/Redpanda, AWS/GCP services, Databases, Caches, File Stores, etc.
• Plugin registry (public API modules, Docker image)
Relationships & interactions
• The CLI initializes the Core Engine, injects the loaded config.
• Core Engine spins up an Input adapter → pushes messages into dispatcher → workers execute Bloblang/processors → dispatch to Output adapter.
• Health checks read adapter states; metrics/traces are emitted throughout.
• Agent/MCP connect to gRPC endpoints inside the Engine.
• Plugin API modules (in “public”) are loaded at startup or via build tags.
Architectural patterns & design principles
• Modularity: each connector in its own package under internal/impl.
• Plug-in architecture: compile-time plugins via build tags; runtime hooks via public package.
• Async pipeline: non-blocking batching and back-pressure (internal/asyncroutine).
• Configuration-driven: declare pipelines in YAML, minimal code changes.
• Sidecar observability: HTTP for liveness/readiness, metrics, tracing.
Relevant technologies & frameworks
• Go modules for dependency management (go.mod)
• gRPC / protobuf for agent & runtime APIs (proto/…)
• Docker multi-stage builds (resources/docker/Dockerfile)
• OpenTelemetry, Prometheus client, Statsd library
• Build system: Makefile, goreleaser, golangci-lint
Detail
• Don’t omit the agent/MCP pieces if they’re relevant to the chosen binary (AI vs Cloud vs Community).
• Plugin API surface under “public/…” to show extensibility points.