anirudha03/spark-redis

Purpose and runtime

Spark-Redis is a Scala connector library that exposes Redis to Apache Spark 3.2.x / Scala 2.12 workloads. It supports Redis standalone and cluster deployments, mapping Redis data structures to Spark RDDs, DataFrames/Spark SQL, DStreams, and Structured Streaming. Build metadata is in pom.xml and project/build.properties.

Public integration entry points

Spark applications enter through implicit/function APIs for RDD and DStream use, plus Spark SQL data-source registration and provider classes:

• RDD APIs: src/main/scala/com/redislabs/provider/redis/redisFunctions.scala, src/main/scala/com/redislabs/provider/redis/rdd/RedisRDD.scala

• DStreams: src/main/scala/com/redislabs/provider/redis/streaming/redisStreamingFunctions.scala, src/main/scala/com/redislabs/provider/redis/streaming/RedisInputDStream.scala

• SQL DataSource V1 registration/provider: src/main/scala/org/apache/spark/sql/redis/DefaultSource.scala, src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister

• Structured Streaming provider: src/main/scala/org/apache/spark/sql/redis/stream/RedisStreamProvider.scala, src/main/scala/org/apache/spark/sql/redis/stream/RedisSource.scala

Connection, configuration, and cluster boundary

RedisConfig centralizes connector options, while ConnectionPool owns reusable Redis connectivity. Connection utilities form the boundary between Spark executors and Redis, including cluster-aware routing/behavior; this boundary must tolerate topology changes described by the connector contract.

• src/main/scala/com/redislabs/provider/redis/RedisConfig.scala

• src/main/scala/com/redislabs/provider/redis/ConnectionPool.scala

• src/main/scala/com/redislabs/provider/redis/util/ConnectionUtils.scala

RDD read/write execution

RedisRDD represents Redis-backed Spark partitions. RedisPartitioner and RedisPartition translate Redis keyspace/cluster distribution into Spark task partitions, so executor tasks can operate near Redis’s slot/node layout. RDD operations flow from Spark transformations/actions through partition-specific Redis reads or writes, using pooled connections.

• src/main/scala/com/redislabs/provider/redis/rdd/RedisRDD.scala

• src/main/scala/com/redislabs/provider/redis/partitioner/RedisPartitioner.scala

• src/main/scala/com/redislabs/provider/redis/partitioner/RedisPartition.scala

Batch SQL/DataFrame persistence

The SQL connector translates DataFrame schemas and rows into Redis persistence models, then exposes reads as a Spark relation. DefaultSource selects relation behavior; RedisSourceRelation handles Spark SQL relation interaction; persistence implementations encode either binary values or Redis hashes.

• src/main/scala/org/apache/spark/sql/redis/DefaultSource.scala

• src/main/scala/org/apache/spark/sql/redis/RedisSourceRelation.scala

• src/main/scala/org/apache/spark/sql/redis/RedisPersistence.scala

• src/main/scala/org/apache/spark/sql/redis/BinaryRedisPersistence.scala

• src/main/scala/org/apache/spark/sql/redis/HashRedisPersistence.scala

Streaming ingestion

Legacy Spark Streaming uses a receiver-backed DStream to pull Redis stream data into micro-batches. Structured Streaming separates source configuration, offsets, reader/RDD creation, and provider integration; offset state is the key control boundary for incremental, restartable processing.

• src/main/scala/com/redislabs/provider/redis/streaming/RedisStreamReceiver.scala

• src/main/scala/org/apache/spark/sql/redis/stream/RedisSourceConfig.scala

• src/main/scala/org/apache/spark/sql/redis/stream/RedisSourceOffset.scala

• src/main/scala/org/apache/spark/sql/redis/stream/RedisStreamReader.scala

• src/main/scala/org/apache/spark/sql/redis/stream/RedisSourceRdd.scala

Operational infrastructure

The artifact is built with Maven/SBT-oriented project files and released through GitHub Actions. Integration CI is defined in .github/workflows/integration.yml; release automation is in .github/workflows/version-and-release.yml. Redis is the external datastore/service, and Spark driver/executors are the consuming runtime.