# rudderlabs/rudder-server architecture

> Architecture diagram of the GitHub repository rudderlabs/rudder-server, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rudderlabs/rudder-server
- Repository: https://github.com/rudderlabs/rudder-server

## Overview

**[Sources] → [Gateway] → [Processor] → [Router] → [Destinations]**
- ↕           ↕            ↕
- [JobsDB] → [Configuration] → [State Store]

##### Key Integration Points

a. External Services
- Control Plane connections
- OAuth services
- Monitoring/debugging services
- Cloud storage services

b. Internal Services
- Transformer service
- Configuration service
- Reporting service
- Stream manager service

##### Deployment Components

a. Core Infrastructure
- Docker containers
- Kubernetes orchestration
- Load balancers
- Service discovery

b. Supporting Services
- Monitoring & alerting
- Logging
- Backup systems
- Security components

##### Additional Elements

a. Performance Components
- Caching layers
- Queue systems
- Batch processing
- Stream processing

b. Monitoring Points
- Health checks
- Metrics collection
- Debugging endpoints
- Performance monitoring

##### The scalability and high availability features

Remember to:

- Use consistent styling and notation
- Clear labels and annotations
- Both synchronous and asynchronous communication paths
- Critical paths and failure points
- Document integration points with external systems

This system design should reflect RudderStack's core capabilities as a CDP while emphasizing its unique features like warehouse-first architecture, developer-focused approach, and extensive integration capabilities.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Data Ingestion Layer"
        GW["Gateway"]:::gateway
        HTTP["HTTP Handlers"]:::gateway
        WH["Webhook Handler"]:::gateway
        AUTH["Authentication"]:::gateway
    end

    subgraph "Processing Layer"
        PROC["Main Processor"]:::processor
        ENRICH["Event Enrichment"]:::processor
        GEO["Geolocation"]:::processor
        TRACK["Tracking Plan"]:::processor
        TRANS["Transformation"]:::processor
    end

    subgraph "Router Layer"
        BATCH["Batch Router"]:::router
        STREAM["Stream Manager"]:::router
        DEST["Destination Manager"]:::router
    end

    subgraph "Storage Layer"
        JOBS[(JobsDB)]:::storage
        CONFIG[(Configuration)]:::storage
        STATE[(State Management)]:::storage
    end

    subgraph "Warehouse Layer"
        WH_MAIN["Main Warehouse"]:::warehouse
        SCHEMA["Schema Management"]:::warehouse
        LOAD["Load Files"]:::warehouse
        INTEG["Warehouse Integrations"]:::warehouse
    end

    %% Data Flow
    GW --> HTTP
    HTTP --> WH
    HTTP --> AUTH
    
    GW --> PROC
    PROC --> ENRICH
    PROC --> GEO
    PROC --> TRACK
    PROC --> TRANS
    
    PROC --> BATCH
    BATCH --> STREAM
    BATCH --> DEST
    
    BATCH --> JOBS
    CONFIG --> PROC
    STATE --> BATCH
    
    DEST --> WH_MAIN
    WH_MAIN --> SCHEMA
    WH_MAIN --> LOAD
    WH_MAIN --> INTEG

    %% Styling
    classDef gateway fill:#e1f5fe,stroke:#01579b
    classDef processor fill:#f3e5f5,stroke:#4a148c
    classDef router fill:#fff3e0,stroke:#e65100
    classDef storage fill:#e8f5e9,stroke:#1b5e20
    classDef warehouse fill:#fce4ec,stroke:#880e4f

    %% Click Events
    click GW "https://github.com/rudderlabs/rudder-server/blob/master/gateway/gateway.go"
    click HTTP "https://github.com/rudderlabs/rudder-server/blob/master/gateway/handle_http.go"
    click WH "https://github.com/rudderlabs/rudder-server/blob/master/gateway/webhook/webhook.go"
    click AUTH "https://github.com/rudderlabs/rudder-server/blob/master/gateway/handle_http_auth.go"
    click PROC "https://github.com/rudderlabs/rudder-server/blob/master/processor/processor.go"
    click ENRICH "https://github.com/rudderlabs/rudder-server/blob/master/internal/enricher/enricher.go"
    click GEO "https://github.com/rudderlabs/rudder-server/blob/master/internal/enricher/geolocation.go"
    click TRACK "https://github.com/rudderlabs/rudder-server/blob/master/processor/trackingplan.go"
    click TRANS "https://github.com/rudderlabs/rudder-server/blob/master/processor/transformer/transformer.go"
    click BATCH "https://github.com/rudderlabs/rudder-server/blob/master/router/batchrouter/batchrouter.go"
    click STREAM "https://github.com/rudderlabs/rudder-server/blob/master/services/streammanager/streammanager.go"
    click DEST "https://github.com/rudderlabs/rudder-server/blob/master/router/batchrouter/asyncdestinationmanager/manager.go"
    click JOBS "https://github.com/rudderlabs/rudder-server/blob/master/jobsdb/jobsdb.go"
    click CONFIG "https://github.com/rudderlabs/rudder-server/blob/master/backend-config/backend-config.go"
    click STATE "https://github.com/rudderlabs/rudder-server/blob/master/app/cluster/state/etcd.go"
    click WH_MAIN "https://github.com/rudderlabs/rudder-server/blob/master/warehouse/app.go"
    click SCHEMA "https://github.com/rudderlabs/rudder-server/blob/master/warehouse/schema/schema.go"
    click LOAD "https://github.com/rudderlabs/rudder-server/blob/master/warehouse/internal/loadfiles/loadfiles.go"
    click INTEG "https://github.com/rudderlabs/rudder-server/tree/master/warehouse/integrations/"
```
