How opensnowcat/opensnowcat-enrich works
Project Type and Purpose
• This is a multi‑module Scala service (“OpenSnowcat Enrich”) for record‑level enrichment of Snowplow events.
• It’s not a UI app but a backend pipeline providing specialized runners for different messaging systems (EventBridge, Kafka, Kinesis, NSQ, PubSub, file).
• Each runner reads raw events, applies a unified FS2‑based enrichment pipeline, then writes enriched or bad records to its target sink.
File Structure Analysis
• Top‑level build.sbt and project/ define the Scala/SBT build.
• config/ holds HOCON files for each runner and JSON schemas for enrichments.
• modules/common-fs2/ contains the core FS2 pipeline, config parsing, I/O clients, telemetry, file and stream sources/sinks.
• modules/common/ holds shared enrichment logic: adapters (raw event parsing), enrichment registry, individual enrichment implementations, loaders, outputs, utilities (Iglu, JSON, HTTP clients).
• modules/cloudutils/ provides cloud‑vendor‑specific clients (AWS S3, Azure Blob, GCP GCS).
• modules/eventbridge, kafka, kinesis, nsq, pubsub each contain:
• src/main – a Main, Source, Sink, Run class wired to common-fs2 and common modules, consuming config/application.conf.
• src/it – integration tests with localstack/Docker Compose and sample config.
• integration-tests/ offers cross‑module smoke tests over common-fs2 with different adapters (SQL, API enrichment).
Key README Insights
• Confirms it’s a fork of Snowplow Enrich focused on “one raw event → one output record.”
Main Components
b. Configuration Loader (reads config/*.hocon, enrichment JSONs, iglu_resolver.json)
c. FS2 Enrichment Pipeline (modules/common-fs2)
d. Enrichment Engine (modules/common – adapters → registry → individual enrichments)
e. Cloud Utils (modules/cloudutils for S3/GCS/Blob if file sinks or enrichment fetches)
f. Output Sinks (writes to respective target: EventBridge, Kafka topic, Kinesis stream, NSQ topic, PubSub subscription)
g. Telemetry & Metrics (FS2 telemetry, StatsDReporter)
h. Iglu Resolver (schema resolution for enrichment JSONs)
Relationships & Data Flow
• Config Loader feeds pipeline parameters into FS2 Enrichment Pipeline.
• Pipeline invokes the Enrichment Engine:
• Adapters parse raw events → pass to Enrichment Registry → each enrichment mutates or validates event → combined output.
• Pipeline writes good/bad records to Output Sink components.
• Cloud Utils clients are invoked by pipeline or enrichment engine for external lookups (e.g., GeoIP DB, S3 asset fetch).
• Telemetry hooks capture metrics at each stage.
Architectural Patterns & Principles
• Hexagonal (Ports & Adapters): Runners and cloudutils act as adapters around a core pipeline.
• Functional Streams (FS2) for back‑pressure and composability.
• Dependency injection via configuration files.
• Monorepo modularization: clear separation of cross‑cutting concerns (common‑fs2 vs common vs runner modules).
Technologies & Frameworks to Label
• Scala, SBT, FS2, Cats Effect
• AWS SDK (EventBridge, DynamoDB, S3), Azure SDK, GCP SDK
• Kafka client, Kinesis client, NSQ client, Google Pub/Sub
• Iglu schema resolver, Circe/Jackson JSON, HOCON (Typesafe Config)