How srikanth-iyengar/datahive works
Project Type and Purpose
• Datahive is essentially a full‑stack, multi‑service data pipeline platform. Its core purpose is to allow users to define and run data transformation workflows using a YAML configuration file.
• It leverages various technologies: Kafka for message streaming, Hadoop for managing large datasets, Spark for data processing (with a custom Spark service), and a user interface for configuring and monitoring pipelines.
Main Components
• Frontend:
– Located in the “app/datahive-frontend” directory.
– Built using Astro (with JavaScript/TypeScript and React components).
– Provides the user interface for pipeline configuration, dashboards, and monitoring (as evidenced by the screenshots in the README).
• Backend/API Service:
– Found under “app/api” (written in Go).
– Provides RESTful or possibly gRPC endpoints (note the proto files in “app/api/pkg/grpc” and service handlers in “app/api/internal/router”).
– Manages pipeline configurations, monitors tasks, and likely orchestrates communication between services.
• Ingestion Service:
– Located in “app/ingestion” (a Java/Spring Boot application).
– Handles receiving data from Kafka and possibly pushes data into Hadoop via its GRPC configuration and working with the proto implementations.
– Plays a role in initiating data flows based on the user’s YAML setups.
• Spark Service:
– Located in “app/spark” (a Java‑based service).
– Manages execution of Apache Spark jobs, uses protobuf based gRPC communication, and includes custom job-related exception handling and utilities.
– Typically used for batch processing jobs.
• Core Library:
– Found in “app/core-lib” which is a Java project.
– Provides shared code and utilities used by one or more backend components, ensuring consistency across the platform.
• Supporting Infrastructure & External Services:
– Kafka: central component for streaming data and job triggers.
– Hadoop: used for managing large-scale batch data necessary for pipeline processing.
– Elasticsearch and Kibana: for logging, visualization, and monitoring of the system.
– Docker and Kubernetes configurations: containerization (“docker” directory, “k8s” directory) for deployments and infrastructure orchestration.
Relationships and Interactions
• The Frontend interacts with the Backend/API service to configure pipelines and display real-time monitoring data.
• The API service communicates with both the Ingestion and Spark services using gRPC/protobuf (as seen in the “pkg/grpc” directories and proto files) to initiate and manage data processing jobs.
• Kafka acts as the messaging backbone connecting various components: the ingestion service listens for messages or sends transformed data, and the Spark service can be triggered based on Kafka events.
• Backend components likely access configuration files (YAML) which define inputs, outputs, and transformation logic that coordinate the data flow across components.
• External systems like Hadoop and Elasticsearch/Kibana provide storage and visualization services, respectively.
Important Architectural Patterns
• Microservices: The project is split into distinct, loosely-coupled services (API, Ingestion, Spark, Core-lib, Frontend) that communicate over standardized protocols.
• Event‑Driven Architecture: Use of Kafka for streaming and triggering events.
• Configuration‑Driven Design: Pipeline behavior and data transformations are defined via YAML, which is consumed by various services.
• Layered Architecture: There is clear separation between user interface, business logic (API, ingestion, spark), and integration with external systems.