# go-graphite/go-carbon architecture

> Architecture diagram of the GitHub repository go-graphite/go-carbon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/go-graphite/go-carbon
- Repository: https://github.com/go-graphite/go-carbon

## Overview

##### Main Components Identification

a. Receivers (Multiple Input Protocols):
- TCP/UDP (Plain text protocol)
- Pickle Protocol
- HTTP
- Kafka
- PubSub
- Protocol Buffers

b. Core Components:
- Cache System
- Persister
- CarbonServer (HTTP API)
- CarbonLink (Query Interface)
- Tags System
- gRPC API

c. Storage:
- Whisper Files
- File System

**[External Clients/Data Sources] → [Input Layer] → [Processing Layer] → [Storage Layer]**
- ↑
- [Query/API Layer]

##### Component Relationships

a. Data Ingestion Flow:
- External sources → Receivers
- Receivers → Cache
- Cache → Persister
- Persister → Whisper Files

b. Query Flow:
- Client Queries → CarbonServer/CarbonLink/gRPC API
- APIs → Cache (for recent data)
- APIs → Whisper Files (for historical data)

##### Detailed Component Specifications

Following details for each component:

a. Cache:
- Max size configuration
- Write strategy options
- Bloom filter capability

b. Persister:
- Multiple worker support
- Whisper file management
- Schema and aggregation handling

c. CarbonServer:
- HTTP API endpoints
- Query cache capabilities
- Index types (Trigram/Trie)

d. Receivers:
- Supported protocols
- Buffer configurations
- Rate limiting options

##### Additional Elements

Supporting elements:

a. Configuration Components:
- Storage schemas
- Storage aggregation
- Storage quotas

b. Monitoring/Metrics:
- Internal statistics
- Prometheus metrics
- Performance metrics

##### Notes Section

Important notes about:
- Scalability considerations
- Performance optimizations
- OS tuning recommendations
- File system requirements

- Multiple input protocol support
- Efficient caching system
- Flexible storage options
- Various query interfaces
- Performance monitoring capabilities

This design accurately represents the project's architecture as shown in the file structure and described in the README, particularly focusing on the core components and their interactions in the Graphite/Carbon server implementation.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph External["External Layer"]
        Clients[Client Applications]:::external
    end

    subgraph Input["Input Layer"]
        direction TB
        TCP["TCP Receiver"]:::receiver
        UDP["UDP Receiver"]:::receiver
        HTTP["HTTP Receiver"]:::receiver
        Kafka["Kafka Receiver"]:::receiver
        PubSub["PubSub Receiver"]:::receiver
        Parser["Protocol Parser"]:::receiver
    end

    subgraph Processing["Processing Layer"]
        direction TB
        Cache["Cache System"]:::core
        CarbonLink["CarbonLink"]:::core
        Persister["Persister"]:::core
        TagSystem["Tags System"]:::core
        Points["Points Handler"]:::core
        PointsReader["Points Reader"]:::core
    end

    subgraph Storage["Storage Layer"]
        direction TB
        WhisperFiles["Whisper Files"]:::storage
        StorageSchemas["Storage Schemas"]:::storage
        StorageAggregation["Storage Aggregation"]:::storage
        StorageQuotas["Storage Quotas"]:::storage
    end

    subgraph API["API Layer"]
        direction TB
        CarbonServer["CarbonServer"]:::api
        GRPC["gRPC API"]:::api
        APIInterface["API Interface"]:::api
    end

    subgraph Core["Core Components"]
        direction TB
        MainApp["Main Application"]:::core
        Config["Configuration"]:::core
        Collector["Collector"]:::core
        Stats["Statistics"]:::core
        ProtoBuf["Protocol Buffers"]:::core
        AtomicOps["Atomic Operations"]:::core
    end

    Clients --> TCP & UDP & HTTP & Kafka & PubSub
    TCP & UDP & HTTP & Kafka & PubSub --> Parser
    Parser --> Points
    Points --> Cache
    Cache --> Persister
    Persister --> WhisperFiles
    TagSystem --> Cache
    CarbonLink --> Cache
    Cache -.-> CarbonServer & GRPC & APIInterface
    WhisperFiles -.-> CarbonServer & GRPC & APIInterface
    StorageSchemas & StorageAggregation & StorageQuotas --> Persister
    MainApp --> Config
    Config --> Collector
    Collector --> Stats
    PointsReader --> Points

    %% Click Events
    click TCP "https://github.com/go-graphite/go-carbon/blob/master/receiver/tcp/tcp.go"
    click UDP "https://github.com/go-graphite/go-carbon/blob/master/receiver/udp/udp.go"
    click HTTP "https://github.com/go-graphite/go-carbon/blob/master/receiver/http/http.go"
    click Kafka "https://github.com/go-graphite/go-carbon/blob/master/receiver/kafka/kafka.go"
    click PubSub "https://github.com/go-graphite/go-carbon/blob/master/receiver/pubsub/pubsub.go"
    click Parser "https://github.com/go-graphite/go-carbon/tree/master/receiver/parse"
    click Cache "https://github.com/go-graphite/go-carbon/blob/master/cache/cache.go"
    click CarbonLink "https://github.com/go-graphite/go-carbon/blob/master/cache/carbonlink.go"
    click Persister "https://github.com/go-graphite/go-carbon/blob/master/persister/whisper.go"
    click TagSystem "https://github.com/go-graphite/go-carbon/blob/master/tags/tags.go"
    click CarbonServer "https://github.com/go-graphite/go-carbon/blob/master/carbonserver/carbonserver.go"
    click APIInterface "https://github.com/go-graphite/go-carbon/blob/master/api/api.go"
    click GRPC "https://github.com/go-graphite/go-carbon/blob/master/helper/grpcutil/grpcutil.go"
    click StorageSchemas "https://github.com/go-graphite/go-carbon/blob/master/deploy/storage-schemas.conf"
    click StorageAggregation "https://github.com/go-graphite/go-carbon/blob/master/deploy/storage-aggregation.conf"
    click StorageQuotas "https://github.com/go-graphite/go-carbon/blob/master/deploy/storage-quota.conf"
    click Points "https://github.com/go-graphite/go-carbon/blob/master/points/points.go"
    click PointsReader "https://github.com/go-graphite/go-carbon/blob/master/points/reader.go"
    click MainApp "https://github.com/go-graphite/go-carbon/blob/master/carbon/app.go"
    click Config "https://github.com/go-graphite/go-carbon/blob/master/carbon/config.go"
    click Collector "https://github.com/go-graphite/go-carbon/blob/master/carbon/collector.go"
    click Stats "https://github.com/go-graphite/go-carbon/blob/master/helper/stat/stat.go"
    click ProtoBuf "https://github.com/go-graphite/go-carbon/blob/master/helper/carbonpb/carbon.pb.go"
    click AtomicOps "https://github.com/go-graphite/go-carbon/blob/master/helper/atomic_utils.go"

    %% Styling
    classDef external fill:#gray,stroke:#333,stroke-width:2px
    classDef receiver fill:#blue,stroke:#333,stroke-width:2px
    classDef core fill:#green,stroke:#333,stroke-width:2px
    classDef storage fill:#yellow,stroke:#333,stroke-width:2px
    classDef api fill:#purple,stroke:#333,stroke-width:2px
```
