# sanketika-obsrv/obsrv-core architecture

> Architecture diagram of the GitHub repository sanketika-obsrv/obsrv-core, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sanketika-obsrv/obsrv-core
- Repository: https://github.com/sanketika-obsrv/obsrv-core

## Overview

##### Project type and purpose
- Obsrv-core is a back-end data-streaming framework built on Apache Flink. It provides a configurable, modular pipeline to extract, validate, enrich, denormalize, route, transform, deduplicate, cache-index and persist streaming data.
- It also includes a Dataset-Registry microservice (CRUD APIs backed by PostgreSQL) for managing metadata and driving parts of the pipeline (e.g., cache-indexer).

##### File structure
- Top-level: CI/CD configs (.github/workflows), Dockerfile, parent pom.xml.
- Modules:
- framework – common Flink utilities (streaming base classes, Redis/Postgres connectors, OTel metrics, SerDe)
- transformation-sdk – pluggable transformer library (Encrypt, Mask, JSONata transformers)
- dataset-registry – microservice + streaming component for metadata (uses Postgres + SQL scripts + Flink streaming registry updates)
- data-products – MasterDataProcessor Flink job (indexes master data)
- pipeline/* – one Flink job per stage: extractor, preprocessor, denormalizer, router, cache-indexer, hudi-connector, transformer, unified-pipeline
- stubs/docker – base Docker images for Flink cluster
- Each job has its own conf, Scala code under src/main/scala/org/sunbird/obsrv, and tests.

##### README for additional insights
- Lists default config for Kafka, Flink checkpointing, Redis, Postgres, job-specific Kafka topics and parallelism
- Documents the Obsrv Core Service REST API (datasets, datasources, queries) – provided by the dataset-registry module

##### Elements
- Clear labels on each component (e.g., “ExtractorJob”, “Kafka Broker”, “Redis Cache”)
· Flink jobs: blue rectangles
· Storage: gray folder/cloud for Hudi/Blob

##### Detail without over-splitting
- Related pipeline stages if desired (e.g., “Core Pipeline”) but list every job
- Both the streaming path and the metadata path (registry → cache-indexer → jobs)
- External dependencies (Kafka, Redis, Postgres, Blob) and shared libs

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources and Brokers
    Kafka((Kafka Cluster)):::kafka

    %% Flink Cluster with Jobs
    subgraph "Flink Cluster"
        direction TB
        ExtractorJob["ExtractorJob"]:::flink
        PreprocessorJob["PreprocessorJob"]:::flink
        DenormalizerJob["DenormalizerJob"]:::flink
        DynamicRouterJob["DynamicRouterJob"]:::flink
        CacheIndexerJob["CacheIndexerJob"]:::flink
        TransformerJob["TransformerJob"]:::flink
        HudiConnectorJob["HudiConnectorJob"]:::flink
        MasterDataProcessorJob["MasterDataProcessorJob"]:::flink
        UnifiedPipelineJob["UnifiedPipelineJob"]:::flink
    end

    %% State Stores and Services
    Redis((Redis Cluster)):::dbgreen
    Postgres((PostgreSQL)):::dbgreen
    DatasetRegistry["Dataset-Registry Service"]:::api
    Hudi[(Hudi Data Lake)]:::storage
    Blob[(Blob Storage)]:::storage

    %% Shared Libraries
    subgraph "Shared Libraries"
        direction TB
        Framework["Framework"]:::lib
        TransformationSDK["Transformation SDK"]:::lib
    end

    %% CI/CD and Project Files
    CI["CI/CD Workflows"]:::lib
    Dockerfile["Root Dockerfile"]:::lib
    Stubs["Flink Cluster Stubs"]:::lib
    POM["Parent pom.xml"]:::lib
    README["README & Configs"]:::lib

    %% Data Flows
    Kafka -->|"ingest topic"| ExtractorJob
    ExtractorJob -->|"raw topic"| PreprocessorJob
    PreprocessorJob -->|"unique topic"| DenormalizerJob
    DenormalizerJob -->|"denorm topic"| DynamicRouterJob
    DynamicRouterJob -->|"routed topic"| TransformerJob
    TransformerJob -->|"transformed topic"| HudiConnectorJob
    HudiConnectorJob -->|writes| Hudi
    HudiConnectorJob -->|stores| Blob

    Kafka -->|"master ingest"| MasterDataProcessorJob

    %% Metadata Path
    DatasetRegistry -->|CRUD APIs| Postgres
    Postgres -->|"registry metadata"| CacheIndexerJob
    Kafka -->|"events"| CacheIndexerJob
    CacheIndexerJob -->|builds cache| Redis

    %% Library Dependencies
    Framework -.-> ExtractorJob
    Framework -.-> PreprocessorJob
    Framework -.-> DenormalizerJob
    Framework -.-> DynamicRouterJob
    Framework -.-> CacheIndexerJob
    Framework -.-> TransformerJob
    Framework -.-> HudiConnectorJob
    Framework -.-> MasterDataProcessorJob
    Framework -.-> UnifiedPipelineJob

    TransformationSDK -.-> TransformerJob

    %% Project Infrastructure
    CI ---|manages| Dockerfile
    CI ---|manages| Stubs
    CI ---|manages| POM
    CI ---|manages| README

    %% Click Events
    click Framework "https://github.com/sanketika-obsrv/obsrv-core/tree/main/framework/"
    click TransformationSDK "https://github.com/sanketika-obsrv/obsrv-core/tree/main/transformation-sdk/"
    click DatasetRegistry "https://github.com/sanketika-obsrv/obsrv-core/tree/main/dataset-registry/"
    click MasterDataProcessorJob "https://github.com/sanketika-obsrv/obsrv-core/tree/main/data-products/"
    click ExtractorJob "https://github.com/sanketika-obsrv/obsrv-core/tree/main/pipeline/extractor/"
    click PreprocessorJob "https://github.com/sanketika-obsrv/obsrv-core/tree/main/pipeline/preprocessor/"
    click DenormalizerJob "https://github.com/sanketika-obsrv/obsrv-core/tree/main/pipeline/denormalizer/"
    click DynamicRouterJob "https://github.com/sanketika-obsrv/obsrv-core/tree/main/pipeline/dataset-router/"
    click CacheIndexerJob "https://github.com/sanketika-obsrv/obsrv-core/tree/main/pipeline/cache-indexer/"
    click TransformerJob "https://github.com/sanketika-obsrv/obsrv-core/tree/main/pipeline/transformer/"
    click HudiConnectorJob "https://github.com/sanketika-obsrv/obsrv-core/tree/main/pipeline/hudi-connector/"
    click UnifiedPipelineJob "https://github.com/sanketika-obsrv/obsrv-core/tree/main/pipeline/unified-pipeline/"
    click CI "https://github.com/sanketika-obsrv/obsrv-core/tree/main/.github/workflows/"
    click Dockerfile "https://github.com/sanketika-obsrv/obsrv-core/tree/main/Dockerfile"
    click Stubs "https://github.com/sanketika-obsrv/obsrv-core/tree/main/stubs/docker/"
    click POM "https://github.com/sanketika-obsrv/obsrv-core/blob/main/pom.xml"
    click README "https://github.com/sanketika-obsrv/obsrv-core/blob/main/README.md"

    %% Styles
    classDef kafka fill:#f39c12,stroke:#e67e22,color:#fff;
    classDef flink fill:#3498db,stroke:#2980b9,color:#fff;
    classDef dbgreen fill:#2ecc71,stroke:#27ae60,color:#fff;
    classDef storage fill:#95a5a6,stroke:#7f8c8d,color:#fff;
    classDef api fill:#9b59b6,stroke:#8e44ad,color:#fff;
    classDef lib fill:#f1c40f,stroke:#f39c12,color:#000;
```
