# srikanth-iyengar/datahive architecture

> Architecture diagram of the GitHub repository srikanth-iyengar/datahive, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/srikanth-iyengar/datahive
- Repository: https://github.com/srikanth-iyengar/datahive

## Overview

##### Project Type and Purpose
- Datahive is essentially a full‑stack, multi‑service data pipeline platform. Its core purpose is to allow users to define and run data transformation workflows using a YAML configuration file.
- It leverages various technologies: Kafka for message streaming, Hadoop for managing large datasets, Spark for data processing (with a custom Spark service), and a user interface for configuring and monitoring pipelines.

##### Main Components
- Frontend:
– Located in the “app/datahive-frontend” directory.
– Built using Astro (with JavaScript/TypeScript and React components).
– Provides the user interface for pipeline configuration, dashboards, and monitoring (as evidenced by the screenshots in the README).

- Backend/API Service:
– Found under “app/api” (written in Go).
– Provides RESTful or possibly gRPC endpoints (note the proto files in “app/api/pkg/grpc” and service handlers in “app/api/internal/router”).
– Manages pipeline configurations, monitors tasks, and likely orchestrates communication between services.

- Ingestion Service:
– Located in “app/ingestion” (a Java/Spring Boot application).
– Handles receiving data from Kafka and possibly pushes data into Hadoop via its GRPC configuration and working with the proto implementations.
– Plays a role in initiating data flows based on the user’s YAML setups.

- Spark Service:
– Located in “app/spark” (a Java‑based service).
– Manages execution of Apache Spark jobs, uses protobuf based gRPC communication, and includes custom job-related exception handling and utilities.
– Typically used for batch processing jobs.

- Core Library:
– Found in “app/core-lib” which is a Java project.
– Provides shared code and utilities used by one or more backend components, ensuring consistency across the platform.

- Supporting Infrastructure & External Services:
– Kafka: central component for streaming data and job triggers.
– Hadoop: used for managing large-scale batch data necessary for pipeline processing.
– Elasticsearch and Kibana: for logging, visualization, and monitoring of the system.
– Docker and Kubernetes configurations: containerization (“docker” directory, “k8s” directory) for deployments and infrastructure orchestration.

##### Relationships and Interactions
- The Frontend interacts with the Backend/API service to configure pipelines and display real-time monitoring data.
- The API service communicates with both the Ingestion and Spark services using gRPC/protobuf (as seen in the “pkg/grpc” directories and proto files) to initiate and manage data processing jobs.
- Kafka acts as the messaging backbone connecting various components: the ingestion service listens for messages or sends transformed data, and the Spark service can be triggered based on Kafka events.
- Backend components likely access configuration files (YAML) which define inputs, outputs, and transformation logic that coordinate the data flow across components.
- External systems like Hadoop and Elasticsearch/Kibana provide storage and visualization services, respectively.

##### Important Architectural Patterns
- Microservices: The project is split into distinct, loosely-coupled services (API, Ingestion, Spark, Core-lib, Frontend) that communicate over standardized protocols.
- Event‑Driven Architecture: Use of Kafka for streaming and triggering events.
- Configuration‑Driven Design: Pipeline behavior and data transformations are defined via YAML, which is consumed by various services.
- Layered Architecture: There is clear separation between user interface, business logic (API, ingestion, spark), and integration with external systems.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        FE["Frontend (Astro, React)"]:::web
    end

    %% Backend Services
    subgraph "Backend Services"
        API["API Service (Go)"]:::backend
        ING["Ingestion Service (Spring Boot, Java)"]:::backend
        SPK["Spark Service (Java)"]:::backend
        CORE["Core Library (Java)"]:::backend
    end

    %% Configuration
    subgraph "Configuration"
        CFG["YAML Pipeline Config"]:::config
    end

    %% Messaging & Storage
    subgraph "Messaging & Storage"
        KAFKA["Kafka (Streaming)"]:::external
        HDD["Hadoop (Batch Storage)"]:::external
        ES["Elasticsearch"]:::external
        KIBANA["Kibana (Dashboard)"]:::external
    end

    %% Infrastructure
    subgraph "Infrastructure"
        DOCK["Docker Config"]:::infra
        KUBE["Kubernetes Config"]:::infra
    end

    %% Connections
    FE -->|"REST/gRPC"| API
    API -->|"gRPC"| ING
    API -->|"gRPC"| SPK
    API -.->|"utilizes"| CORE
    ING -.->|"utilizes"| CORE
    SPK -.->|"utilizes"| CORE

    API -.->|"uses"| CFG
    ING -.->|"uses"| CFG
    SPK -.->|"uses"| CFG

    ING -->|"pub/sub"| KAFKA
    SPK -->|"job triggers"| KAFKA
    SPK -->|"writes-to"| HDD

    API -->|"fetches"| ES
    FE -->|"displays"| KIBANA

    %% Click Events
    click FE "https://github.com/srikanth-iyengar/datahive/tree/master/app/datahive-frontend"
    click API "https://github.com/srikanth-iyengar/datahive/tree/master/app/api"
    click ING "https://github.com/srikanth-iyengar/datahive/tree/master/app/ingestion"
    click SPK "https://github.com/srikanth-iyengar/datahive/tree/master/app/spark"
    click CORE "https://github.com/srikanth-iyengar/datahive/tree/master/app/core-lib"
    click DOCK "https://github.com/srikanth-iyengar/datahive/tree/master/docker"
    click KUBE "https://github.com/srikanth-iyengar/datahive/tree/master/k8s"

    %% Styles
    classDef web fill:#a9d18e,stroke:#4f6228,stroke-width:2px;
    classDef backend fill:#ffd966,stroke:#bf9000,stroke-width:2px;
    classDef external fill:#9fc5e8,stroke:#1155cc,stroke-width:2px;
    classDef infra fill:#f4cccc,stroke:#a64d79,stroke-width:2px;
    classDef config fill:#d9d2e9,stroke:#9673a6,stroke-width:2px;
```
