How rangareddy/ranga_spark_experiments works
This repository is a collection of Apache Spark experiments, not one end-to-end application. Its examples demonstrate batch integrations with storage and query systems, Kafka and structured streaming, RDD operations, Spark launching, and graceful streaming shutdown. Each example is independently initiated; the principal workflow is to create Spark data or consume a stream, apply an example transformation, and display or write results. The README identifies several integrations, while most source excerpts are absent, so links between unsampled examples and external systems are documented only where the README supports them.
Main entry points: SparkStreamingKuduExample.scala Kafka to Kudu · spark_gcs_integration GCS examples · shutdown Graceful shutdown · StopByHttpHandler.scala HTTP stop handler
Components 26
Storage integrations
- GCS examples
spark_gcs_integration - GCS Scala example
spark-gcs-integration/src/main/scala/com/ranga/spark/gcs/SparkGcsIntegrationApp.scala - S3 example
spark_s3_integration - HBase examples
spark-hbase-integration - HBase CDH
spark-hbase-cdh-integration/src/main/scala/com/ranga/spark/hbase/cdh/SparkHbaseCdhIntegrationApp.scala - HBase SHC CDH
spark-hbase-shc-cdh-integration/src/main/scala/com/ranga/spark/hbase/shc/cdh/SparkHbaseShcCdhIntegrationApp.scala - Hive HBase
spark-hive-hbase-integration/src/main/scala/com/ranga/spark/hive/hbase/SparkHiveHbaseIntegrationApp.scala - Kudu examples
spark-kudu-integration - File formats
spark-avro-integration - Other stores
spark-cassandra-integration
Kafka and streaming
- Kafka examples
spark-kafka-integration/src/main/scala/com/ranga/spark/kafka/SparkKafkaIntegrationApp.scala - Structured streaming
spark-kafka-structured-streaming-integration/src/main/scala/com/ranga/spark/kafka/structured/streaming/SparkKafkaStructuredStreamingIntegrationApp.scala - Secure Kafka
spark3_kafka_integration/src/main/scala/com/ranga/spark/kafka/SparkKafkaSecureStructuredStreaming.scala - Streaming Kafka
spark-streaming-kafka-integration/src/main/scala/com/ranga/spark/streaming/kafka/SparkStreamingKafkaIntegrationApp.scala - Kafka to Kudu
spark_kudu_integration/SparkStreamingKuduExample.scala - Spark 3 Kafka
spark3-kafka-integration/src/main/scala/com/ranga/spark3/kafka/Spark3KafkaIntegrationApp.scala
Spark examples
- RDD operations
spark_rdd_operations/src/main/scala/com/ranga/spark/rdd - Hello world
spark-hello-world-integration/src/main/scala/com/ranga/spark/hello/world/SparkHelloWorldIntegrationApp.scala - Spark launcher
spark-launcher-integration/src/main/scala/com/ranga/spark/launcher/SparkLauncherIntegrationApp.scala
Streaming operations
- Graceful shutdown
spark-streaming-graceful-shutdown/src/main/scala/com/ranga/spark/streaming/shutdown - HTTP stop handler
spark-streaming-graceful-shutdown/src/main/scala/com/ranga/spark/streaming/shutdown/util/http/StopByHttpHandler.scala
Other components
- Example runner
- Kafka
- Google Cloud Storage
- Storage systems
- Spark runtime
Connections 25
- Example runner → GCS examples: runs
spark_gcs_integration/src/main/scala/com/ranga/spark/gcs/SparkGCSIntegrationExample.scala - GCS examples → Google Cloud Storage: reads and writes
spark_gcs_integration/src/main/scala/com/ranga/spark/gcs/SparkGCSIntegrationExample.scala - Example runner → GCS Scala example: runs
- Example runner → S3 example: runs
- Example runner → HBase examples: runs
- Example runner → HBase CDH: runs
- Example runner → HBase SHC CDH: runs
- Example runner → Hive HBase: runs
- Example runner → Kudu examples: runs
- Example runner → File formats: runs
- Example runner → Other stores: runs
- Example runner → Kafka examples: runs
- Example runner → Structured streaming: runs
- Example runner → Secure Kafka: runs
- Example runner → Streaming Kafka: runs
- Example runner → Kafka to Kudu: runs
spark_kudu_integration/SparkStreamingKuduExample.scala - Kafka to Kudu → Kafka: consumes
spark_kudu_integration/SparkStreamingKuduExample.scala - Kafka to Kudu → Storage systems: writes
spark_kudu_integration/SparkStreamingKuduExample.scala - Example runner → Spark 3 Kafka: runs
- Example runner → RDD operations: runs
- Example runner → Hello world: runs
- Example runner → Spark launcher: runs
- Example runner → Graceful shutdown: runs
- Graceful shutdown → HTTP stop handler: uses
- HTTP stop handler → Spark runtime: stops gracefully
spark-streaming-graceful-shutdown/src/main/scala/com/ranga/spark/streaming/shutdown/util/http/StopByHttpHandler.scala