# rangareddy/ranga_spark_experiments architecture

> Architecture diagram of the GitHub repository rangareddy/ranga_spark_experiments, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rangareddy/ranga_spark_experiments
- Repository: https://github.com/rangareddy/ranga_spark_experiments

## Overview

This repository is a collection of Apache Spark experiments, not one end-to-end application. Its examples demonstrate batch integrations with storage and query systems, Kafka and structured streaming, RDD operations, Spark launching, and graceful streaming shutdown. Each example is independently initiated; the principal workflow is to create Spark data or consume a stream, apply an example transformation, and display or write results. The README identifies several integrations, while most source excerpts are absent, so links between unsampled examples and external systems are documented only where the README supports them.

Main entry points: [`spark_kudu_integration/SparkStreamingKuduExample.scala`](https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark_kudu_integration/SparkStreamingKuduExample.scala) (Kafka to Kudu), [`spark_gcs_integration`](https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark_gcs_integration) (GCS examples), [`spark-streaming-graceful-shutdown/src/main/scala/com/ranga/spark/streaming/shutdown`](https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark-streaming-graceful-shutdown/src/main/scala/com/ranga/spark/streaming/shutdown) (Graceful shutdown), [`spark-streaming-graceful-shutdown/src/main/scala/com/ranga/spark/streaming/shutdown/util/http/StopByHttpHandler.scala`](https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-streaming-graceful-shutdown/src/main/scala/com/ranga/spark/streaming/shutdown/util/http/StopByHttpHandler.scala) (HTTP stop handler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_storage["Storage integrations"]
  node_gcs["GCS examples"]
  node_gcslegacy["GCS Scala example"]
  node_s3["S3 example"]
  node_hbase["HBase examples"]
  node_hbasecdh["HBase CDH"]
  node_shc["HBase SHC CDH"]
  node_hivehbase["Hive HBase"]
  node_kudu["Kudu examples"]
  node_formats["File formats"]
  node_otherstores["Other stores"]
end

subgraph group_streaming["Kafka and streaming"]
  node_kafka["Kafka examples"]
  node_structured["Structured streaming"]
  node_securekafka["Secure Kafka"]
  node_streamingkafka["Streaming Kafka"]
  node_kudu_stream["Kafka to Kudu"]
  node_kafka3["Spark 3 Kafka"]
end

subgraph group_spark["Spark examples"]
  node_rdd["RDD operations"]
  node_hello["Hello world"]
  node_launcher["Spark launcher"]
end

subgraph group_operations["Streaming operations"]
  node_shutdown["Graceful shutdown"]
  node_httpstop["HTTP stop handler"]
end

node_user(("Example runner"))
node_kafkaexternal["Kafka"]
node_gcsservice[("Google Cloud Storage")]
node_hadoopstores[("Storage systems")]
node_spark["Spark runtime"]

node_user -->|"runs"| node_gcs
node_gcs -->|"reads and writes"| node_gcsservice
node_user -.->|"runs"| node_gcslegacy
node_user -.->|"runs"| node_s3
node_user -.->|"runs"| node_hbase
node_user -.->|"runs"| node_hbasecdh
node_user -.->|"runs"| node_shc
node_user -.->|"runs"| node_hivehbase
node_user -.->|"runs"| node_kudu
node_user -.->|"runs"| node_formats
node_user -.->|"runs"| node_otherstores
node_user -.->|"runs"| node_kafka
node_user -.->|"runs"| node_structured
node_user -.->|"runs"| node_securekafka
node_user -.->|"runs"| node_streamingkafka
node_user -->|"runs"| node_kudu_stream
node_kudu_stream -->|"consumes"| node_kafkaexternal
node_kudu_stream -->|"writes"| node_hadoopstores
node_user -.->|"runs"| node_kafka3
node_user -.->|"runs"| node_rdd
node_user -.->|"runs"| node_hello
node_user -.->|"runs"| node_launcher
node_user -.->|"runs"| node_shutdown
node_shutdown -.->|"uses"| node_httpstop
node_httpstop -->|"stops gracefully"| node_spark

click node_gcs "https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark_gcs_integration"
click node_gcslegacy "https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-gcs-integration/src/main/scala/com/ranga/spark/gcs/SparkGcsIntegrationApp.scala"
click node_s3 "https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark_s3_integration"
click node_hbase "https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark-hbase-integration"
click node_hbasecdh "https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-hbase-cdh-integration/src/main/scala/com/ranga/spark/hbase/cdh/SparkHbaseCdhIntegrationApp.scala"
click node_shc "https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-hbase-shc-cdh-integration/src/main/scala/com/ranga/spark/hbase/shc/cdh/SparkHbaseShcCdhIntegrationApp.scala"
click node_hivehbase "https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-hive-hbase-integration/src/main/scala/com/ranga/spark/hive/hbase/SparkHiveHbaseIntegrationApp.scala"
click node_kudu "https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark-kudu-integration"
click node_formats "https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark-avro-integration"
click node_otherstores "https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark-cassandra-integration"
click node_kafka "https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-kafka-integration/src/main/scala/com/ranga/spark/kafka/SparkKafkaIntegrationApp.scala"
click node_structured "https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-kafka-structured-streaming-integration/src/main/scala/com/ranga/spark/kafka/structured/streaming/SparkKafkaStructuredStreamingIntegrationApp.scala"
click node_securekafka "https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark3_kafka_integration/src/main/scala/com/ranga/spark/kafka/SparkKafkaSecureStructuredStreaming.scala"
click node_streamingkafka "https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-streaming-kafka-integration/src/main/scala/com/ranga/spark/streaming/kafka/SparkStreamingKafkaIntegrationApp.scala"
click node_kudu_stream "https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark_kudu_integration/SparkStreamingKuduExample.scala"
click node_kafka3 "https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark3-kafka-integration/src/main/scala/com/ranga/spark3/kafka/Spark3KafkaIntegrationApp.scala"
click node_rdd "https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark_rdd_operations/src/main/scala/com/ranga/spark/rdd"
click node_hello "https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-hello-world-integration/src/main/scala/com/ranga/spark/hello/world/SparkHelloWorldIntegrationApp.scala"
click node_launcher "https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-launcher-integration/src/main/scala/com/ranga/spark/launcher/SparkLauncherIntegrationApp.scala"
click node_shutdown "https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark-streaming-graceful-shutdown/src/main/scala/com/ranga/spark/streaming/shutdown"
click node_httpstop "https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-streaming-graceful-shutdown/src/main/scala/com/ranga/spark/streaming/shutdown/util/http/StopByHttpHandler.scala"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_gcs,node_gcslegacy,node_s3,node_hbase,node_hbasecdh,node_shc,node_hivehbase,node_kudu,node_formats,node_otherstores toneBlue
class node_kafka,node_structured,node_securekafka,node_streamingkafka,node_kudu_stream,node_kafka3,node_gcsservice,node_hadoopstores toneAmber
class node_rdd,node_hello,node_launcher toneMint
class node_shutdown,node_httpstop toneRose
class node_user,node_kafkaexternal,node_spark toneIndigo
```

## Components

### Storage integrations

- [GCS examples](https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark_gcs_integration): `spark_gcs_integration`
- [GCS Scala example](https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-gcs-integration/src/main/scala/com/ranga/spark/gcs/SparkGcsIntegrationApp.scala): `spark-gcs-integration/src/main/scala/com/ranga/spark/gcs/SparkGcsIntegrationApp.scala`
- [S3 example](https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark_s3_integration): `spark_s3_integration`
- [HBase examples](https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark-hbase-integration): `spark-hbase-integration`
- [HBase CDH](https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-hbase-cdh-integration/src/main/scala/com/ranga/spark/hbase/cdh/SparkHbaseCdhIntegrationApp.scala): `spark-hbase-cdh-integration/src/main/scala/com/ranga/spark/hbase/cdh/SparkHbaseCdhIntegrationApp.scala`
- [HBase SHC CDH](https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-hbase-shc-cdh-integration/src/main/scala/com/ranga/spark/hbase/shc/cdh/SparkHbaseShcCdhIntegrationApp.scala): `spark-hbase-shc-cdh-integration/src/main/scala/com/ranga/spark/hbase/shc/cdh/SparkHbaseShcCdhIntegrationApp.scala`
- [Hive HBase](https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-hive-hbase-integration/src/main/scala/com/ranga/spark/hive/hbase/SparkHiveHbaseIntegrationApp.scala): `spark-hive-hbase-integration/src/main/scala/com/ranga/spark/hive/hbase/SparkHiveHbaseIntegrationApp.scala`
- [Kudu examples](https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark-kudu-integration): `spark-kudu-integration`
- [File formats](https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark-avro-integration): `spark-avro-integration`
- [Other stores](https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark-cassandra-integration): `spark-cassandra-integration`

### Kafka and streaming

- [Kafka examples](https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-kafka-integration/src/main/scala/com/ranga/spark/kafka/SparkKafkaIntegrationApp.scala): `spark-kafka-integration/src/main/scala/com/ranga/spark/kafka/SparkKafkaIntegrationApp.scala`
- [Structured streaming](https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-kafka-structured-streaming-integration/src/main/scala/com/ranga/spark/kafka/structured/streaming/SparkKafkaStructuredStreamingIntegrationApp.scala): `spark-kafka-structured-streaming-integration/src/main/scala/com/ranga/spark/kafka/structured/streaming/SparkKafkaStructuredStreamingIntegrationApp.scala`
- [Secure Kafka](https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark3_kafka_integration/src/main/scala/com/ranga/spark/kafka/SparkKafkaSecureStructuredStreaming.scala): `spark3_kafka_integration/src/main/scala/com/ranga/spark/kafka/SparkKafkaSecureStructuredStreaming.scala`
- [Streaming Kafka](https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-streaming-kafka-integration/src/main/scala/com/ranga/spark/streaming/kafka/SparkStreamingKafkaIntegrationApp.scala): `spark-streaming-kafka-integration/src/main/scala/com/ranga/spark/streaming/kafka/SparkStreamingKafkaIntegrationApp.scala`
- [Kafka to Kudu](https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark_kudu_integration/SparkStreamingKuduExample.scala): `spark_kudu_integration/SparkStreamingKuduExample.scala`
- [Spark 3 Kafka](https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark3-kafka-integration/src/main/scala/com/ranga/spark3/kafka/Spark3KafkaIntegrationApp.scala): `spark3-kafka-integration/src/main/scala/com/ranga/spark3/kafka/Spark3KafkaIntegrationApp.scala`

### Spark examples

- [RDD operations](https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark_rdd_operations/src/main/scala/com/ranga/spark/rdd): `spark_rdd_operations/src/main/scala/com/ranga/spark/rdd`
- [Hello world](https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-hello-world-integration/src/main/scala/com/ranga/spark/hello/world/SparkHelloWorldIntegrationApp.scala): `spark-hello-world-integration/src/main/scala/com/ranga/spark/hello/world/SparkHelloWorldIntegrationApp.scala`
- [Spark launcher](https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-launcher-integration/src/main/scala/com/ranga/spark/launcher/SparkLauncherIntegrationApp.scala): `spark-launcher-integration/src/main/scala/com/ranga/spark/launcher/SparkLauncherIntegrationApp.scala`

### Streaming operations

- [Graceful shutdown](https://github.com/rangareddy/ranga_spark_experiments/tree/master/spark-streaming-graceful-shutdown/src/main/scala/com/ranga/spark/streaming/shutdown): `spark-streaming-graceful-shutdown/src/main/scala/com/ranga/spark/streaming/shutdown`
- [HTTP stop handler](https://github.com/rangareddy/ranga_spark_experiments/blob/master/spark-streaming-graceful-shutdown/src/main/scala/com/ranga/spark/streaming/shutdown/util/http/StopByHttpHandler.scala): `spark-streaming-graceful-shutdown/src/main/scala/com/ranga/spark/streaming/shutdown/util/http/StopByHttpHandler.scala`

### Other components

- **Example runner**
- **Kafka**
- **Google Cloud Storage**
- **Storage systems**
- **Spark runtime**

## Connections

- Example runner → GCS examples: runs (evidence: `spark_gcs_integration/src/main/scala/com/ranga/spark/gcs/SparkGCSIntegrationExample.scala`)
- GCS examples → Google Cloud Storage: reads and writes (evidence: `spark_gcs_integration/src/main/scala/com/ranga/spark/gcs/SparkGCSIntegrationExample.scala`)
- Example runner → GCS Scala example: runs
- Example runner → S3 example: runs
- Example runner → HBase examples: runs
- Example runner → HBase CDH: runs
- Example runner → HBase SHC CDH: runs
- Example runner → Hive HBase: runs
- Example runner → Kudu examples: runs
- Example runner → File formats: runs
- Example runner → Other stores: runs
- Example runner → Kafka examples: runs
- Example runner → Structured streaming: runs
- Example runner → Secure Kafka: runs
- Example runner → Streaming Kafka: runs
- Example runner → Kafka to Kudu: runs (evidence: `spark_kudu_integration/SparkStreamingKuduExample.scala`)
- Kafka to Kudu → Kafka: consumes (evidence: `spark_kudu_integration/SparkStreamingKuduExample.scala`)
- Kafka to Kudu → Storage systems: writes (evidence: `spark_kudu_integration/SparkStreamingKuduExample.scala`)
- Example runner → Spark 3 Kafka: runs
- Example runner → RDD operations: runs
- Example runner → Hello world: runs
- Example runner → Spark launcher: runs
- Example runner → Graceful shutdown: runs
- Graceful shutdown → HTTP stop handler: uses
- HTTP stop handler → Spark runtime: stops gracefully (evidence: `spark-streaming-graceful-shutdown/src/main/scala/com/ranga/spark/streaming/shutdown/util/http/StopByHttpHandler.scala`)