# warpstreamlabs/bento architecture

> Architecture diagram of the GitHub repository warpstreamlabs/bento, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/warpstreamlabs/bento
- Repository: https://github.com/warpstreamlabs/bento

## Overview

##### Main Components Architecture

**┌─────────────────────────────────────────── Bento Service ───────────────────────────────────────────┐**

- ┌─────────────┐    ┌───────────────┐    ┌────────────────┐    ┌──────────────┐    ┌────────────┐  │
- Inputs    │ -> │   Pipeline    │ -> │   Processors   │ -> │   Buffers    │ -> │  Outputs   │  │
- ┘    └───────────────┘    └────────────────┘    └──────────────┘    └────────────┘  │
- ↑                   │                     │                    │                   ↓         │

- ┌─────────────────────────────────────── Service Layer ──────────────────────────────────────┐     │

- ┌──────────┐    ┌──────────┐    ┌───────────┐    ┌──────────┐    ┌────────────────┐     │     │
- Cache   │    │ Metrics  │    │  Tracing  │    │  Config  │    │ Rate Limiting  │     │     │
- ┘    └──────────┘    └───────────┘    └──────────┘    └────────────────┘     │     │
- ┘     │
- ┘

##### Component Details

a. Input/Output Connectors:
**┌─ Supported Connectors ─────┐**
- • AWS (DynamoDB, S3, etc) │
- • Azure Services          │
- • GCP Services           │
- • Kafka                  │
- • NATS                   │
- • Redis                  │
- • Databases             │
- • HTTP/WebSocket        │
- ┘

b. Processing Pipeline:
**┌─ Pipeline Flow ──────────────────────────────────────┐**

- Message → Transform → Enrich → Filter → Route       │
- ↑          │         │         │        │       │
- ┴─────────┴─────────┴────────┘       │

- ┘

##### Observability Layer

**┌─ Observability ────────────────────────┐**

- ┌─────────┐  ┌────────┐  ┌────────┐  │
- Metrics │  │ Traces │  │ Logs   │  │
- ┘  └────────┘  └────────┘  │

- ┌─────────────────────────────────┐  │
- Monitoring Endpoints        │  │
- • /ping (Liveness)            │  │
- • /ready (Readiness)          │  │
- ┘  │
- ┘

##### Configuration Management
**┌─ Configuration ──────────────────┐**

- ┌──────────┐    ┌───────────┐  │
- YAML    │ -> │ Resource  │  │
- Config  │    │ Manager   │  │
- ┘    └───────────┘  │

- ┘

##### Important Implementation Details

Technical aspects:
- Transaction model for at-least-once delivery
- Serverless deployment options
- Plugin architecture
- Resource management
- Error handling flows

- → Message Flow             │
- ⇢ Configuration Flow       │
- ⟳ Retry/Recovery Path      │
- ■ Core Component           │
- □ Optional Component       │
- ┘

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph BentoService["Bento Service"]
        subgraph CoreLayer["Core Processing Layer"]
            Inputs["Input Connectors"]:::input
            Pipeline["Pipeline Framework"]:::pipeline
            Processors["Processors"]:::processor
            Buffers["Buffers"]:::buffer
            Outputs["Output Connectors"]:::output

            Inputs --> Pipeline
            Pipeline --> Processors
            Processors --> Buffers
            Buffers --> Outputs
        end

        subgraph ServiceLayer["Service Layer"]
            Cache["Cache Service"]:::service
            Metrics["Metrics Service"]:::service
            Tracing["Tracing Service"]:::service
            Config["Configuration"]:::service
            RateLimit["Rate Limiting"]:::service
        end

        subgraph ObservabilityLayer["Observability"]
            MetricsImpl["Metrics Implementation"]:::obs
            TracingImpl["Tracing Implementation"]:::obs
            Logging["Logging System"]:::obs
        end

        subgraph ConfigManagement["Configuration Management"]
            CoreConfig["Core Config"]:::config
            Schema["Config Schema"]:::config
            ResourceMgr["Resource Manager"]:::config
        end

        subgraph PluginSystem["Plugin System"]
            PluginFramework["Plugin Framework"]:::plugin
            PluginImpl["Plugin Implementations"]:::plugin
        end

        subgraph ErrorHandling["Error Handling"]
            ErrorFramework["Error Framework"]:::error
            TransactionHandler["Transaction Handler"]:::error
        end
    end

    %% Relationships
    ServiceLayer --> CoreLayer
    ObservabilityLayer --> CoreLayer
    ConfigManagement --> CoreLayer
    PluginSystem --> CoreLayer
    ErrorHandling --> CoreLayer

    %% Click Events
    click Pipeline "https://github.com/warpstreamlabs/bento/tree/main//internal/pipeline/"
    click Processors "https://github.com/warpstreamlabs/bento/tree/main//internal/component/processor/"
    click Inputs "https://github.com/warpstreamlabs/bento/tree/main//internal/component/input/"
    click Outputs "https://github.com/warpstreamlabs/bento/tree/main//internal/component/output/"
    click Buffers "https://github.com/warpstreamlabs/bento/tree/main//internal/component/buffer/"
    click Cache "https://github.com/warpstreamlabs/bento/tree/main//internal/component/cache/"
    click Metrics "https://github.com/warpstreamlabs/bento/tree/main//internal/component/metrics/"
    click Tracing "https://github.com/warpstreamlabs/bento/tree/main//internal/component/tracer/"
    click RateLimit "https://github.com/warpstreamlabs/bento/tree/main//internal/component/ratelimit/"
    click Config "https://github.com/warpstreamlabs/bento/tree/main//internal/config/"
    click MetricsImpl "https://github.com/warpstreamlabs/bento/tree/main//internal/component/metrics/"
    click TracingImpl "https://github.com/warpstreamlabs/bento/tree/main//internal/tracing/"
    click Logging "https://github.com/warpstreamlabs/bento/tree/main//internal/log/"
    click CoreConfig "https://github.com/warpstreamlabs/bento/tree/main//internal/config/"
    click Schema "https://github.com/warpstreamlabs/bento/tree/main//internal/config/schema/"
    click ResourceMgr "https://github.com/warpstreamlabs/bento/tree/main//internal/manager/"
    click PluginFramework "https://github.com/warpstreamlabs/bento/tree/main//public/components/"
    click PluginImpl "https://github.com/warpstreamlabs/bento/tree/main//internal/impl/"
    click ErrorFramework "https://github.com/warpstreamlabs/bento/tree/main//internal/errorhandling/"
    click TransactionHandler "https://github.com/warpstreamlabs/bento/tree/main//internal/transaction/"

    %% Styles
    classDef input fill:#90EE90
    classDef output fill:#90EE90
    classDef pipeline fill:#87CEEB
    classDef processor fill:#87CEEB
    classDef buffer fill:#87CEEB
    classDef service fill:#D3D3D3
    classDef obs fill:#FFD700
    classDef config fill:#A9A9A9
    classDef plugin fill:#DDA0DD
    classDef error fill:#FFB6C1

    %% Legend
    subgraph Legend
        L1["Input/Output"]:::input
        L2["Core Processing"]:::pipeline
        L3["Services"]:::service
        L4["Observability"]:::obs
        L5["Configuration"]:::config
        L6["Plugins"]:::plugin
        L7["Error Handling"]:::error
    end
```
