# newscred/webhook-broker architecture

> Architecture diagram of the GitHub repository newscred/webhook-broker, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/newscred/webhook-broker
- Repository: https://github.com/newscred/webhook-broker

## Overview

##### Main Components

a. Core Components:
- HTTP API Layer (controllers/)
- Message Dispatcher System (dispatcher/)
- Storage Layer (storage/)
- Configuration Management (config/)
- Message Pruning System (prune/)
- Database Migration System (migration/)

b. External Components:
- Producer Services
- Consumer Services
- Relational Database
- Message Store

##### Component Relationships

- Producer → HTTP API → Message Storage → Dispatcher → Consumer
- Configuration → All Components
- Database Migrations → Database
- Pruning Service → Message Store

##### Specific Implementation Details

**┌─────────────────────────────────────────────────────────────┐**
- External Services                        │
- (Producer Services)              (Consumer Services)       │
- ┘
- ↓                            ↑
**┌─────────────────────────────────────────────────────────────┐**
- HTTP API Layer                           │
- Broadcast Controller    - Channel Controller             │
- Consumer Controller     - Producer Controller            │
- Message Controller      - Status Controller              │
- ┘
- ↓                            ↑
**┌─────────────────────────────────────────────────────────────┐**
- Message Dispatcher                          │
- Queue Management                                         │
- Worker Pool                                             │
- Delivery Management                                      │
- ┘
- ↓                            ↑
**┌─────────────────────────────────────────────────────────────┐**
- Storage Layer                           │
- Channel Repository    - Consumer Repository              │
- Producer Repository   - Message Repository               │
- Delivery Job Repository                                  │
- ┘
- ↓                            ↑
**┌─────────────────────────────────────────────────────────────┐**
- Database & Message Store                    │
- ┘

##### Important Elements to Highlight

a. Message Flow:
- Complete path of a message from producer to consumer
- Retry mechanisms and error handling
- Message pruning flow

b. Configuration Management:
- How configuration affects each component
- Dynamic configuration capabilities

c. Deployment Components:
- Kubernetes/Helm deployment options
- High-availability setup
- Load balancing components

##### Technical Details

- Database migrations flow
- Message pruning mechanism
- Lock management system
- Cache layer (memcache)
- Health check endpoints
- Metrics collection points

Remember to include deployment-related components from the Helm charts, showing how the system can be deployed in a Kubernetes environment with proper scaling and high availability.

This design should reflect the project's goal of providing a reliable, scalable pub/sub broker system for microservice architectures while maintaining simplicity in implementation and usage.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph External
        Producer["Producer Services"]:::external
        Consumer["Consumer Services"]:::external
    end

    subgraph APILayer["HTTP API Layer"]
        BroadcastCtrl["Broadcast Controller"]:::api
        ChannelCtrl["Channel Controller"]:::api
        ConsumerCtrl["Consumer Controller"]:::api
        ProducerCtrl["Producer Controller"]:::api
        MessageCtrl["Message Controller"]:::api
        StatusCtrl["Status Controller"]:::api
    end

    subgraph DispatcherSystem["Message Dispatcher System"]
        QueueMgmt["Queue Management"]:::dispatcher
        WorkerPool["Worker Pool"]:::dispatcher
        MsgDispatcher["Message Dispatcher"]:::dispatcher
    end

    subgraph StorageLayer["Storage Layer"]
        ChannelRepo["Channel Repository"]:::storage
        ConsumerRepo["Consumer Repository"]:::storage
        ProducerRepo["Producer Repository"]:::storage
        MessageRepo["Message Repository"]:::storage
        DeliveryJobRepo["Delivery Job Repository"]:::storage
    end

    subgraph ConfigSystem["Configuration Management"]
        MainConfig["Main Config"]:::config
        APIConfig["API Config"]:::config
        DefaultConfig["Default Config"]:::config
    end

    subgraph PruneSystem["Message Pruning System"]
        ArchiveWriter["Archive Writer"]:::prune
        PruneService["Pruning Service"]:::prune
    end

    subgraph DBSystem["Database System"]
        DB[(Database)]:::db
        Migrations["DB Migrations"]:::db
    end

    Producer --> BroadcastCtrl & MessageCtrl
    BroadcastCtrl & MessageCtrl --> MsgDispatcher
    MsgDispatcher --> WorkerPool
    WorkerPool --> Consumer

    QueueMgmt --> WorkerPool
    MsgDispatcher --> QueueMgmt

    StorageLayer --> DB
    Migrations --> DB

    DispatcherSystem --> StorageLayer
    APILayer --> StorageLayer

    PruneSystem --> StorageLayer
    ConfigSystem --> APILayer & DispatcherSystem & StorageLayer & PruneSystem

    %% Click Events
    click BroadcastCtrl "https://github.com/newscred/webhook-broker/blob/main/controllers/broadcast.go"
    click ChannelCtrl "https://github.com/newscred/webhook-broker/blob/main/controllers/channel.go"
    click ConsumerCtrl "https://github.com/newscred/webhook-broker/blob/main/controllers/consumer.go"
    click ProducerCtrl "https://github.com/newscred/webhook-broker/blob/main/controllers/producer.go"
    click MessageCtrl "https://github.com/newscred/webhook-broker/blob/main/controllers/message.go"
    click StatusCtrl "https://github.com/newscred/webhook-broker/blob/main/controllers/status.go"
    
    click QueueMgmt "https://github.com/newscred/webhook-broker/blob/main/dispatcher/queue.go"
    click WorkerPool "https://github.com/newscred/webhook-broker/blob/main/dispatcher/worker.go"
    click MsgDispatcher "https://github.com/newscred/webhook-broker/blob/main/dispatcher/msgdispatcher.go"
    
    click ChannelRepo "https://github.com/newscred/webhook-broker/blob/main/storage/channelrepo.go"
    click ConsumerRepo "https://github.com/newscred/webhook-broker/blob/main/storage/consumerrepo.go"
    click ProducerRepo "https://github.com/newscred/webhook-broker/blob/main/storage/producerrepo.go"
    click MessageRepo "https://github.com/newscred/webhook-broker/blob/main/storage/messagerepo.go"
    click DeliveryJobRepo "https://github.com/newscred/webhook-broker/blob/main/storage/deliveryjobrepo.go"
    
    click MainConfig "https://github.com/newscred/webhook-broker/blob/main/config/config.go"
    click APIConfig "https://github.com/newscred/webhook-broker/blob/main/config/api.go"
    click DefaultConfig "https://github.com/newscred/webhook-broker/blob/main/config/defaultconfig.go"
    
    click ArchiveWriter "https://github.com/newscred/webhook-broker/blob/main/prune/archivewriter.go"
    click PruneService "https://github.com/newscred/webhook-broker/blob/main/prune/service.go"
    
    click Migrations "https://github.com/newscred/webhook-broker/tree/main/migration/sqls"

    %% Styles
    classDef external fill:#a8d1f7
    classDef api fill:#90EE90
    classDef dispatcher fill:#FFB347
    classDef storage fill:#FFE5B4
    classDef config fill:#E6E6FA
    classDef prune fill:#FFA07A
    classDef db fill:#D3D3D3

    %% Legend
    subgraph Legend
        ExtLegend["External Services"]:::external
        APILegend["API Layer"]:::api
        DispLegend["Dispatcher"]:::dispatcher
        StoreLegend["Storage"]:::storage
        ConfigLegend["Configuration"]:::config
        PruneLegend["Pruning"]:::prune
        DBLegend["Database"]:::db
    end
```
