# anirudha03/spark-redis architecture

> Architecture diagram of the GitHub repository anirudha03/spark-redis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anirudha03/spark-redis
- Repository: https://github.com/anirudha03/spark-redis

## Overview

#### Purpose and runtime
Spark-Redis is a Scala connector library that exposes Redis to Apache Spark 3.2.x / Scala 2.12 workloads. It supports Redis standalone and cluster deployments, mapping Redis data structures to Spark RDDs, DataFrames/Spark SQL, DStreams, and Structured Streaming. Build metadata is in `pom.xml` and `project/build.properties`.

#### Public integration entry points
Spark applications enter through implicit/function APIs for RDD and DStream use, plus Spark SQL data-source registration and provider classes:
- RDD APIs: `src/main/scala/com/redislabs/provider/redis/redisFunctions.scala`, `src/main/scala/com/redislabs/provider/redis/rdd/RedisRDD.scala`
- DStreams: `src/main/scala/com/redislabs/provider/redis/streaming/redisStreamingFunctions.scala`, `src/main/scala/com/redislabs/provider/redis/streaming/RedisInputDStream.scala`
- SQL DataSource V1 registration/provider: `src/main/scala/org/apache/spark/sql/redis/DefaultSource.scala`, `src/main/resources/META-INF/services/org.apache.spark.sql.sources.DataSourceRegister`
- Structured Streaming provider: `src/main/scala/org/apache/spark/sql/redis/stream/RedisStreamProvider.scala`, `src/main/scala/org/apache/spark/sql/redis/stream/RedisSource.scala`

#### Connection, configuration, and cluster boundary
`RedisConfig` centralizes connector options, while `ConnectionPool` owns reusable Redis connectivity. Connection utilities form the boundary between Spark executors and Redis, including cluster-aware routing/behavior; this boundary must tolerate topology changes described by the connector contract.
- `src/main/scala/com/redislabs/provider/redis/RedisConfig.scala`
- `src/main/scala/com/redislabs/provider/redis/ConnectionPool.scala`
- `src/main/scala/com/redislabs/provider/redis/util/ConnectionUtils.scala`

#### RDD read/write execution
`RedisRDD` represents Redis-backed Spark partitions. `RedisPartitioner` and `RedisPartition` translate Redis keyspace/cluster distribution into Spark task partitions, so executor tasks can operate near Redis’s slot/node layout. RDD operations flow from Spark transformations/actions through partition-specific Redis reads or writes, using pooled connections.
- `src/main/scala/com/redislabs/provider/redis/rdd/RedisRDD.scala`
- `src/main/scala/com/redislabs/provider/redis/partitioner/RedisPartitioner.scala`
- `src/main/scala/com/redislabs/provider/redis/partitioner/RedisPartition.scala`

#### Batch SQL/DataFrame persistence
The SQL connector translates DataFrame schemas and rows into Redis persistence models, then exposes reads as a Spark relation. `DefaultSource` selects relation behavior; `RedisSourceRelation` handles Spark SQL relation interaction; persistence implementations encode either binary values or Redis hashes.
- `src/main/scala/org/apache/spark/sql/redis/DefaultSource.scala`
- `src/main/scala/org/apache/spark/sql/redis/RedisSourceRelation.scala`
- `src/main/scala/org/apache/spark/sql/redis/RedisPersistence.scala`
- `src/main/scala/org/apache/spark/sql/redis/BinaryRedisPersistence.scala`
- `src/main/scala/org/apache/spark/sql/redis/HashRedisPersistence.scala`

#### Streaming ingestion
Legacy Spark Streaming uses a receiver-backed DStream to pull Redis stream data into micro-batches. Structured Streaming separates source configuration, offsets, reader/RDD creation, and provider integration; offset state is the key control boundary for incremental, restartable processing.
- `src/main/scala/com/redislabs/provider/redis/streaming/RedisStreamReceiver.scala`
- `src/main/scala/org/apache/spark/sql/redis/stream/RedisSourceConfig.scala`
- `src/main/scala/org/apache/spark/sql/redis/stream/RedisSourceOffset.scala`
- `src/main/scala/org/apache/spark/sql/redis/stream/RedisStreamReader.scala`
- `src/main/scala/org/apache/spark/sql/redis/stream/RedisSourceRdd.scala`

#### Operational infrastructure
The artifact is built with Maven/SBT-oriented project files and released through GitHub Actions. Integration CI is defined in `.github/workflows/integration.yml`; release automation is in `.github/workflows/version-and-release.yml`. Redis is the external datastore/service, and Spark driver/executors are the consuming runtime.

Main entry points: [`src/main/scala/com/redislabs/provider/redis/ConnectionPool.scala`](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/ConnectionPool.scala) (Cluster-aware connections), [`src/main/scala/org/apache/spark/sql/redis/stream/RedisSource.scala`](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/stream/RedisSource.scala) (Structured Redis source), [`src/main/scala/com/redislabs/provider/redis/rdd/RedisRDD.scala`](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/rdd/RedisRDD.scala) (RedisRDD), [`src/main/scala/org/apache/spark/sql/redis/RedisSourceRelation.scala`](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/RedisSourceRelation.scala) (RedisSourceRelation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Spark integration"]
  node_rdd_api["RDD functions<br/>implicit API"]
  node_dstream_api["DStream functions<br/>implicit API"]
end

subgraph group_core["Connector execution"]
  node_redis_rdd["RedisRDD<br/>Spark RDD<br/>[RedisRDD.scala]"]
  node_config["RedisConfig<br/>connector options<br/>[RedisConfig.scala]"]
  node_partitioning["Redis partition mapping<br/>partition planner"]
  node_connections["Cluster-aware connections<br/>connection boundary"]
end

subgraph group_sql["Spark SQL"]
  node_sql_provider["SQL DefaultSource<br/>DataSource V1 provider"]
  node_sql_relation["RedisSourceRelation<br/>SQL relation"]
  node_persistence["Redis persistence models<br/>row encoder"]
end

subgraph group_streaming["Streaming"]
  node_legacy_stream["Redis input DStream<br/>receiver DStream"]
  node_stream_receiver["Redis stream receiver"]
  node_structured_provider["Structured stream provider"]
  node_structured_source["Structured Redis source<br/>micro-batch source<br/>[RedisSource.scala]"]
  node_source_config["Source config<br/>stream configuration"]
  node_offsets["Source offsets<br/>checkpoint state"]
  node_stream_reader["Stream reader<br/>reader factory"]
  node_source_rdd["Redis source RDD<br/>stream read RDD"]
end

subgraph group_ops["Build &amp; delivery"]
  node_build["Build &amp; release automation<br/>delivery infrastructure<br/>[pom.xml]"]
end

node_spark{{"Spark driver &amp; executors<br/>external runtime"}}
node_redis[("Redis standalone / cluster<br/>external datastore")]

node_spark -->|"uses"| node_rdd_api
node_rdd_api -->|"creates"| node_redis_rdd
node_redis_rdd -->|"plans tasks"| node_partitioning
node_partitioning -->|"routes partitions"| node_connections
node_redis_rdd -->|"reads / writes"| node_connections
node_spark -->|"uses"| node_dstream_api
node_dstream_api -->|"creates"| node_legacy_stream
node_legacy_stream -->|"runs"| node_stream_receiver
node_stream_receiver -->|"consumes"| node_connections
node_spark -->|"loads SQL source"| node_sql_provider
node_sql_provider -->|"selects relation"| node_sql_relation
node_sql_relation -->|"encodes rows"| node_persistence
node_sql_relation -->|"reads / writes"| node_connections
node_spark -->|"loads stream source"| node_structured_provider
node_structured_provider -->|"creates"| node_structured_source
node_structured_source -->|"uses"| node_source_config
node_structured_source -->|"tracks progress"| node_offsets
node_structured_source -->|"plans batches"| node_stream_reader
node_stream_reader -->|"creates"| node_source_rdd
node_source_rdd -->|"reads"| node_connections
node_config -->|"configures"| node_connections
node_connections -->|"pooled cluster-aware access"| node_redis
node_build -.->|"publishes connector artifact"| node_spark

click node_rdd_api "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/redisFunctions.scala"
click node_redis_rdd "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/rdd/RedisRDD.scala"
click node_dstream_api "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/streaming/redisStreamingFunctions.scala"
click node_legacy_stream "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/streaming/RedisInputDStream.scala"
click node_stream_receiver "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/streaming/RedisStreamReceiver.scala"
click node_sql_provider "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/DefaultSource.scala"
click node_sql_relation "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/RedisSourceRelation.scala"
click node_persistence "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/RedisPersistence.scala"
click node_structured_provider "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/stream/RedisStreamProvider.scala"
click node_structured_source "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/stream/RedisSource.scala"
click node_source_config "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/stream/RedisSourceConfig.scala"
click node_offsets "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/stream/RedisSourceOffset.scala"
click node_stream_reader "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/stream/RedisStreamReader.scala"
click node_source_rdd "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/stream/RedisSourceRdd.scala"
click node_config "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/RedisConfig.scala"
click node_partitioning "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/partitioner/RedisPartitioner.scala"
click node_connections "https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/ConnectionPool.scala"
click node_build "https://github.com/anirudha03/spark-redis/blob/master/pom.xml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_rdd_api,node_dstream_api toneBlue
class node_redis_rdd,node_config,node_partitioning,node_connections toneAmber
class node_sql_provider,node_sql_relation,node_persistence toneMint
class node_legacy_stream,node_stream_receiver,node_structured_provider,node_structured_source,node_source_config,node_offsets,node_stream_reader,node_source_rdd toneRose
class node_build toneIndigo
class node_spark,node_redis toneNeutral
```

## Components

### Spark integration

- [RDD functions](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/redisFunctions.scala): implicit API · `src/main/scala/com/redislabs/provider/redis/redisFunctions.scala`
- [DStream functions](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/streaming/redisStreamingFunctions.scala): implicit API · `src/main/scala/com/redislabs/provider/redis/streaming/redisStreamingFunctions.scala`

### Connector execution

- [RedisRDD](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/rdd/RedisRDD.scala): Spark RDD · Redis-backed partitioned read/write execution. · `src/main/scala/com/redislabs/provider/redis/rdd/RedisRDD.scala`
- [RedisConfig](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/RedisConfig.scala): connector options · `src/main/scala/com/redislabs/provider/redis/RedisConfig.scala`
- [Redis partition mapping](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/partitioner/RedisPartitioner.scala): partition planner · Maps keyspace and cluster distribution to Spark task partitions. · `src/main/scala/com/redislabs/provider/redis/partitioner/RedisPartitioner.scala`
- [Cluster-aware connections](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/ConnectionPool.scala): connection boundary · Pools executor connections and handles cluster-aware routing. · `src/main/scala/com/redislabs/provider/redis/ConnectionPool.scala`

### Spark SQL

- [SQL DefaultSource](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/DefaultSource.scala): DataSource V1 provider · Registered through the Spark SQL service entry. · `src/main/scala/org/apache/spark/sql/redis/DefaultSource.scala`
- [RedisSourceRelation](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/RedisSourceRelation.scala): SQL relation · Executes batch DataFrame reads and writes. · `src/main/scala/org/apache/spark/sql/redis/RedisSourceRelation.scala`
- [Redis persistence models](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/RedisPersistence.scala): row encoder · Binary and hash persistence implementations encode DataFrame rows. · `src/main/scala/org/apache/spark/sql/redis/RedisPersistence.scala`

### Streaming

- [Redis input DStream](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/streaming/RedisInputDStream.scala): receiver DStream · Builds receiver-backed micro-batches. · `src/main/scala/com/redislabs/provider/redis/streaming/RedisInputDStream.scala`
- [Redis stream receiver](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/com/redislabs/provider/redis/streaming/RedisStreamReceiver.scala): stream receiver · `src/main/scala/com/redislabs/provider/redis/streaming/RedisStreamReceiver.scala`
- [Structured stream provider](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/stream/RedisStreamProvider.scala): stream provider · `src/main/scala/org/apache/spark/sql/redis/stream/RedisStreamProvider.scala`
- [Structured Redis source](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/stream/RedisSource.scala): micro-batch source · Coordinates incremental source execution. · `src/main/scala/org/apache/spark/sql/redis/stream/RedisSource.scala`
- [Source config](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/stream/RedisSourceConfig.scala): stream configuration · `src/main/scala/org/apache/spark/sql/redis/stream/RedisSourceConfig.scala`
- [Source offsets](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/stream/RedisSourceOffset.scala): checkpoint state · Controls restartable incremental reads. · `src/main/scala/org/apache/spark/sql/redis/stream/RedisSourceOffset.scala`
- [Stream reader](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/stream/RedisStreamReader.scala): reader factory · Creates stream read work. · `src/main/scala/org/apache/spark/sql/redis/stream/RedisStreamReader.scala`
- [Redis source RDD](https://github.com/anirudha03/spark-redis/blob/master/src/main/scala/org/apache/spark/sql/redis/stream/RedisSourceRdd.scala): stream read RDD · `src/main/scala/org/apache/spark/sql/redis/stream/RedisSourceRdd.scala`

### Build & delivery

- [Build & release automation](https://github.com/anirudha03/spark-redis/blob/master/pom.xml): delivery infrastructure · Maven/SBT project metadata and GitHub Actions integration and release workflows. · `pom.xml`

### Other components

- **Spark driver & executors**: external runtime · Consumer runtime for connector APIs and task execution.
- **Redis standalone / cluster**: external datastore · Target datastore with cluster topology changes.

## Connections

- Spark driver & executors → RDD functions: uses
- RDD functions → RedisRDD: creates
- RedisRDD → Redis partition mapping: plans tasks
- Redis partition mapping → Cluster-aware connections: routes partitions
- RedisRDD → Cluster-aware connections: reads / writes
- Spark driver & executors → DStream functions: uses
- DStream functions → Redis input DStream: creates
- Redis input DStream → Redis stream receiver: runs
- Redis stream receiver → Cluster-aware connections: consumes
- Spark driver & executors → SQL DefaultSource: loads SQL source
- SQL DefaultSource → RedisSourceRelation: selects relation
- RedisSourceRelation → Redis persistence models: encodes rows
- RedisSourceRelation → Cluster-aware connections: reads / writes
- Spark driver & executors → Structured stream provider: loads stream source
- Structured stream provider → Structured Redis source: creates
- Structured Redis source → Source config: uses
- Structured Redis source → Source offsets: tracks progress
- Structured Redis source → Stream reader: plans batches
- Stream reader → Redis source RDD: creates
- Redis source RDD → Cluster-aware connections: reads
- RedisConfig → Cluster-aware connections: configures
- Cluster-aware connections → Redis standalone / cluster: pooled cluster-aware access
- Build & release automation → Spark driver & executors: publishes connector artifact