# opensnowcat/opensnowcat-enrich architecture

> Architecture diagram of the GitHub repository opensnowcat/opensnowcat-enrich, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opensnowcat/opensnowcat-enrich
- Repository: https://github.com/opensnowcat/opensnowcat-enrich

## Overview

##### Project Type and Purpose
- This is a multi‑module Scala service (“OpenSnowcat Enrich”) for record‑level enrichment of Snowplow events.
- It’s not a UI app but a backend pipeline providing specialized runners for different messaging systems (EventBridge, Kafka, Kinesis, NSQ, PubSub, file).
- Each runner reads raw events, applies a unified FS2‑based enrichment pipeline, then writes enriched or bad records to its target sink.

##### File Structure Analysis
- Top‑level build.sbt and project/ define the Scala/SBT build.
- config/ holds HOCON files for each runner and JSON schemas for enrichments.
- modules/common-fs2/ contains the core FS2 pipeline, config parsing, I/O clients, telemetry, file and stream sources/sinks.
- modules/common/ holds shared enrichment logic: adapters (raw event parsing), enrichment registry, individual enrichment implementations, loaders, outputs, utilities (Iglu, JSON, HTTP clients).
- modules/cloudutils/ provides cloud‑vendor‑specific clients (AWS S3, Azure Blob, GCP GCS).
- modules/eventbridge, kafka, kinesis, nsq, pubsub each contain:
- src/main – a Main, Source, Sink, Run class wired to common-fs2 and common modules, consuming config/application.conf.
- src/it – integration tests with localstack/Docker Compose and sample config.
- integration-tests/ offers cross‑module smoke tests over common-fs2 with different adapters (SQL, API enrichment).

##### Key README Insights
- Confirms it’s a fork of Snowplow Enrich focused on “one raw event → one output record.”

##### Main Components
b. Configuration Loader (reads config/*.hocon, enrichment JSONs, iglu_resolver.json)
c. FS2 Enrichment Pipeline (modules/common-fs2)
d. Enrichment Engine (modules/common – adapters → registry → individual enrichments)
e. Cloud Utils (modules/cloudutils for S3/GCS/Blob if file sinks or enrichment fetches)
f. Output Sinks (writes to respective target: EventBridge, Kafka topic, Kinesis stream, NSQ topic, PubSub subscription)
g. Telemetry & Metrics (FS2 telemetry, StatsDReporter)
h. Iglu Resolver (schema resolution for enrichment JSONs)

##### Relationships & Data Flow
- Config Loader feeds pipeline parameters into FS2 Enrichment Pipeline.
- Pipeline invokes the Enrichment Engine:
- Adapters parse raw events → pass to Enrichment Registry → each enrichment mutates or validates event → combined output.
- Pipeline writes good/bad records to Output Sink components.
- Cloud Utils clients are invoked by pipeline or enrichment engine for external lookups (e.g., GeoIP DB, S3 asset fetch).
- Telemetry hooks capture metrics at each stage.

##### Architectural Patterns & Principles
- Hexagonal (Ports & Adapters): Runners and cloudutils act as adapters around a core pipeline.
- Functional Streams (FS2) for back‑pressure and composability.
- Dependency injection via configuration files.
- Monorepo modularization: clear separation of cross‑cutting concerns (common‑fs2 vs common vs runner modules).

##### Technologies & Frameworks to Label
- Scala, SBT, FS2, Cats Effect
- AWS SDK (EventBridge, DynamoDB, S3), Azure SDK, GCP SDK
- Kafka client, Kinesis client, NSQ client, Google Pub/Sub
- Iglu schema resolver, Circe/Jackson JSON, HOCON (Typesafe Config)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Source Runners"
        EB["EventBridgeSource"]:::source
        KA["KafkaSource"]:::source
        KI["KinesisSource"]:::source
        NS["NSQSource"]:::source
        PU["PubSubSource"]:::source
        FI["FileSource"]:::source
    end

    CL["Configuration & Enrichment JSONs"]:::config
    IG["IgluResolver"]:::config

    subgraph "Core Pipeline"
        EP["EnrichPipeline (FS2 | Cats Effect)"]:::core
        EN["Enrichment Engine"]:::core
    end

    subgraph "Output Sinks"
        EBS["EventBridgeSink"]:::sink
        KAS["KafkaSink"]:::sink
        KIS["KinesisSink"]:::sink
        NSS["NSQSink"]:::sink
        PUS["PubSubSink"]:::sink
    end

    CU["Cloud Utils (AWS/Azure/GCP)"]:::cloud
    TM["Telemetry & Metrics"]:::telemetry

    EB -->|"raw events"| CL
    KA -->|""raw events""| CL
    KI -->|""raw events""| CL
    NS -->|""raw events""| CL
    PU -->|""raw events""| CL
    FI -->|""raw events""| CL

    CL -->|"config params"| EP
    IG -->|"schema resolution"| CL
    IG -->|"schema resolution"| EN

    EP -->|"invoke enrichments"| EN
    EP -->|"enriched/bad records"| EBS
    EP -->|"enriched/bad records"| KAS
    EP -->|"enriched/bad records"| KIS
    EP -->|"enriched/bad records"| NSS
    EP -->|"enriched/bad records"| PUS
    EP -->|"metrics hooks"| TM

    EP -.->|""external lookups""| CU
    EN -.->|""external lookups""| CU

    classDef source fill:#a8e6cf,stroke:#379634,color:#000
    classDef config fill:#dcedc1,stroke:#8fb996,color:#000
    classDef core fill:#aedff7,stroke:#2196f3,color:#000
    classDef sink fill:#ffd3b6,stroke:#ff8c00,color:#000
    classDef cloud fill:#e0e0e0,stroke:#9e9e9e,color:#000
    classDef telemetry fill:#e1bee7,stroke:#9c27b0,color:#000

    click EB "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/modules/eventbridge/src/main/scala/com/snowplowanalytics/snowplow/enrich/eventbridge/Source.scala"
    click KA "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/modules/kafka/src/main/scala/com/snowplowanalytics/snowplow.enrich.kafka/Source.scala"
    click KI "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/modules/kinesis/src/main/scala/com/snowplowanalytics/snowplow/enrich/kinesis/Source.scala"
    click NS "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/modules/nsq/src/main/scala/com/snowplowanalytics/snowplow/enrich/nsq/Source.scala"
    click PU "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/modules/pubsub/src/main/scala/com/snowplowanalytics/snowplow/enrich/pubsub/Source.scala"
    click FI "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/modules/common-fs2/src/main/scala/com/snowplowanalytics/snowplow/enrich/common/fs2/io/FileSystem.scala"
    click CL "https://github.com/opensnowcat/opensnowcat-enrich/tree/main/config/"
    click IG "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/config/iglu_resolver.json"
    click EP "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/modules/common-fs2/src/main/scala/com/snowplowanalytics/snowplow/enrich/common/fs2/Enrich.scala"
    click EN "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/modules/common/src/main/scala/com/snowplowanalytics/snowplow.enrich/common/adapters/AdapterRegistry.scala"
    click EBS "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/modules/eventbridge/src/main/scala/com/snowplowanalytics/snowplow/enrich/eventbridge/Sink.scala"
    click KAS "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/modules/kafka/src/main/scala/com.snowplowanalytics.snowplow.enrich.kafka/Sink.scala"
    click KIS "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/modules/kinesis/src/main/scala/com/snowplowanalytics/snowplow/enrich/kinesis/Sink.scala"
    click NSS "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/modules/nsq/src/main/scala/com/snowplowanalytics/snowplow/enrich/nsq/Sink.scala"
    click PUS "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/modules/pubsub/src/main/scala/com/snowplowanalytics/snowplow/enrich/pubsub/Sink.scala"
    click CU "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/modules/cloudutils/aws/src/main/scala/com/snowplowanalytics/snowplow/enrich/aws/S3Client.scala"
    click TM "https://github.com/opensnowcat/opensnowcat-enrich/blob/main/modules/common-fs2/src/main/scala/com/snowplowanalytics/snowplow/enrich/common/fs2/Telemetry.scala"
```
