How sanketika-obsrv/obsrv-core works
Project type and purpose
• Obsrv-core is a back-end data-streaming framework built on Apache Flink. It provides a configurable, modular pipeline to extract, validate, enrich, denormalize, route, transform, deduplicate, cache-index and persist streaming data.
• It also includes a Dataset-Registry microservice (CRUD APIs backed by PostgreSQL) for managing metadata and driving parts of the pipeline (e.g., cache-indexer).
File structure
• Top-level: CI/CD configs (.github/workflows), Dockerfile, parent pom.xml.
• Modules:
• framework – common Flink utilities (streaming base classes, Redis/Postgres connectors, OTel metrics, SerDe)
• transformation-sdk – pluggable transformer library (Encrypt, Mask, JSONata transformers)
• dataset-registry – microservice + streaming component for metadata (uses Postgres + SQL scripts + Flink streaming registry updates)
• data-products – MasterDataProcessor Flink job (indexes master data)
• pipeline/* – one Flink job per stage: extractor, preprocessor, denormalizer, router, cache-indexer, hudi-connector, transformer, unified-pipeline
• stubs/docker – base Docker images for Flink cluster
• Each job has its own conf, Scala code under src/main/scala/org/sunbird/obsrv, and tests.
README for additional insights
• Lists default config for Kafka, Flink checkpointing, Redis, Postgres, job-specific Kafka topics and parallelism
• Documents the Obsrv Core Service REST API (datasets, datasources, queries) – provided by the dataset-registry module
Elements
• Clear labels on each component (e.g., “ExtractorJob”, “Kafka Broker”, “Redis Cache”)
· Flink jobs: blue rectangles
· Storage: gray folder/cloud for Hudi/Blob
Detail without over-splitting
• Related pipeline stages if desired (e.g., “Core Pipeline”) but list every job
• Both the streaming path and the metadata path (registry → cache-indexer → jobs)
• External dependencies (Kafka, Redis, Postgres, Blob) and shared libs