# opensnowcat/opensnowcat-collector architecture

> Architecture diagram of the GitHub repository opensnowcat/opensnowcat-collector, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opensnowcat/opensnowcat-collector
- Repository: https://github.com/opensnowcat/opensnowcat-collector

## Overview

##### Main Components Identification

The system should be represented with these key components:

a. Core Collector Service
- HTTP Server (Akka-based)
- Request Processing Layer
- Analytics.js Bridge
- Health Check Service
- Telemetry Service

b. Sink Adapters (Multiple Output Options)
- Kinesis
- Kafka
- Google PubSub
- RabbitMQ
- NSQ
- SQS
- Stdout

**[Client Requests] → [Load Balancer/HTTP Endpoint]**

**[Collector Service]**
- Request Handler
- Analytics.js Bridge
- Health Check Service
- Telemetry Service

**[Sink Interface]**
- Kinesis Sink
- Kafka Sink
- PubSub Sink
- RabbitMQ Sink
- NSQ Sink
- SQS Sink
- Stdout Sink

##### Important Details

a. Configuration Layer:
- Configuration management (HOCON files)
- Environment-specific settings
- Fallback mechanisms

b. Testing Components:
- Integration test framework
- Test sinks
- Monitoring endpoints

##### Specific Implementation Notes

Architectural patterns:
- Actor-based concurrency (Akka)
- Stream processing patterns
- Sink abstraction layer
- Health check mechanisms
- Telemetry data flow

##### External Dependencies

Connections to:
- Cloud services (AWS, GCP)
- Message brokers
- Monitoring systems
- Analytics platforms

##### Deployment Context

Include:
- Container boundaries
- Scaling points
- High availability setup
- Load balancing points

##### Scalability points

This collector is part of a larger data pipeline, so indicate the boundaries and interfaces where it connects to other systems in the data flow.

## Diagram (Mermaid)

```mermaid
graph TB
    %% External Layer
    Client(["Client Requests"]):::external
    LB["Load Balancer"]:::external

    %% Core Processing Layer
    subgraph Core["Core Processing"]
        CollectorService["Collector Service"]:::core
        RequestHandler["Request Processing Layer"]:::core
        AnalyticsBridge["Analytics.js Bridge"]:::core
        HealthCheck["Health Check Service"]:::core
        Telemetry["Telemetry Service"]:::monitoring
    end

    %% Sink Interface Layer
    subgraph Sinks["Sink Layer"]
        SinkInterface["Sink Interface"]:::sink
        KinesisSink["Kinesis Sink"]:::sink
        KafkaSink["Kafka Sink"]:::sink
        PubSubSink["Google PubSub Sink"]:::sink
        RabbitMQSink["RabbitMQ Sink"]:::sink
        NSQSink["NSQ Sink"]:::sink
        SQSSink["SQS Sink"]:::sink
        StdoutSink["Stdout Sink"]:::sink
    end

    %% External Services
    subgraph ExternalServices["External Services"]
        AWS["AWS Services"]:::external
        Kafka["Kafka Cluster"]:::external
        PubSub["Google PubSub"]:::external
        RabbitMQ["RabbitMQ"]:::external
        NSQ["NSQ"]:::external
    end

    %% Connections
    Client --> LB
    LB --> CollectorService
    CollectorService --> RequestHandler
    CollectorService --> AnalyticsBridge
    CollectorService --> HealthCheck
    CollectorService --> Telemetry
    RequestHandler --> SinkInterface
    SinkInterface --> KinesisSink
    SinkInterface --> KafkaSink
    SinkInterface --> PubSubSink
    SinkInterface --> RabbitMQSink
    SinkInterface --> NSQSink
    SinkInterface --> SQSSink
    SinkInterface --> StdoutSink
    KinesisSink -.-> AWS
    KafkaSink -.-> Kafka
    PubSubSink -.-> PubSub
    RabbitMQSink -.-> RabbitMQ
    NSQSink -.-> NSQ
    SQSSink -.-> AWS

    %% Click Events
    click CollectorService "https://github.com/opensnowcat/opensnowcat-collector/blob/main/core/src/main/scala/com.snowplowanalytics.snowplow.collectors.scalastream/CollectorService.scala"
    click AnalyticsBridge "https://github.com/opensnowcat/opensnowcat-collector/blob/main/core/src/main/scala/com.snowplowanalytics.snowplow.collectors.scalastream/AnalyticsJsBridge.scala"
    click HealthCheck "https://github.com/opensnowcat/opensnowcat-collector/blob/main/core/src/main/scala/com.snowplowanalytics.snowplow.collectors.scalastream/HealthService.scala"
    click Telemetry "https://github.com/opensnowcat/opensnowcat-collector/blob/main/core/src/main/scala/com.snowplowanalytics.snowplow.collectors.scalastream/telemetry/TelemetryAkkaService.scala"
    click SinkInterface "https://github.com/opensnowcat/opensnowcat-collector/blob/main/core/src/main/scala/com.snowplowanalytics.snowplow.collectors.scalastream/sinks/Sink.scala"
    click KinesisSink "https://github.com/opensnowcat/opensnowcat-collector/blob/main/kinesis/src/main/scala/com.snowplowanalytics.snowplow.collectors.scalastream/sinks/KinesisSink.scala"
    click KafkaSink "https://github.com/opensnowcat/opensnowcat-collector/blob/main/kafka/src/main/scala/com.snowplowanalytics.snowplow.collectors.scalastream/sinks/KafkaSink.scala"
    click PubSubSink "https://github.com/opensnowcat/opensnowcat-collector/blob/main/pubsub/src/main/scala/com.snowplowanalytics.snowplow.collectors.scalastream/sinks/GooglePubSubSink.scala"
    click RabbitMQSink "https://github.com/opensnowcat/opensnowcat-collector/blob/main/rabbitmq/src/main/scala/com.snowplowanalytics.snowplow.collectors.scalastream/sinks/RabbitMQSink.scala"
    click NSQSink "https://github.com/opensnowcat/opensnowcat-collector/blob/main/nsq/src/main/scala/com.snowplowanalytics.snowplow.collectors.scalastream/sinks/NsqSink.scala"
    click SQSSink "https://github.com/opensnowcat/opensnowcat-collector/blob/main/sqs/src/main/scala/com.snowplowanalytics.snowplow.collectors.scalastream/sinks/SqsSink.scala"
    click StdoutSink "https://github.com/opensnowcat/opensnowcat-collector/blob/main/stdout/src/main/scala/com.snowplowanalytics.snowplow.collectors.scalastream/sinks/StdoutSink.scala"
    click RequestHandler "https://github.com/opensnowcat/opensnowcat-collector/blob/main/core/src/main/scala/com.snowplowanalytics.snowplow.collectors.scalastream/CollectorRoute.scala"

    %% Styles
    classDef external fill:#A0CBE8
    classDef core fill:#8FBC8F
    classDef sink fill:#FFAB40
    classDef monitoring fill:#B39DDB
    
    %% Legend
    subgraph Legend
        External["External Component"]:::external
        CoreComp["Core Component"]:::core
        SinkComp["Sink Component"]:::sink
        MonitoringComp["Monitoring Component"]:::monitoring
    end
```
