How nvidia/cudf-spark works
The repository is NVIDIA’s cuDF plugin for Apache Spark: Spark workloads are accelerated on GPUs while aiming for Spark-compatible results. The main runtime story is Spark invoking the plugin, which initializes GPU resources and provides an optimized shuffle manager; Delta Lake support extends this to table operations and transaction-aware writes. The tree documents broad Delta capabilities, but the supplied source excerpts mainly cover shuffle, resource management, and compatibility shims, so detailed wiring among Delta command implementations is not asserted.
Main entry points: GpuOptimisticTransaction.scala Transactions · RapidsShuffleInternalManagerBase.scala Shuffle manager · GpuDeleteCommand.scala DML commands · GpuMergeIntoCommand.scala Merge commands
Components 22
Spark runtime
- GPU acceleration
- Partition joins
sql-plugin/src/main/spark400/scala/com/nvidia/spark/rapids/shims/StoragePartitionJoinShims.scala
GPU resources
- GPU manager
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuDeviceManager.scala
Shuffle transport
- Shuffle proxy
sql-plugin-api/src/main/scala/org/apache/spark/sql/rapids/ProxyRapidsShuffleInternalManagerBase.scala - Shuffle manager
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/RapidsShuffleInternalManagerBase.scala - Version manager
sql-plugin/src/main/spark330/scala/com/nvidia/spark/rapids/RapidsShuffleManager.scala - Bounce buffers
sql-plugin/src/main/scala/com/nvidia/spark/rapids/shuffle/BounceBufferManager.scala - Async throttling
sql-plugin/src/main/scala/com/nvidia/spark/rapids/io/async/TrafficController.scala
Delta Lake
- Delta provider
delta-lake/common/src/main/delta-33x-42x/scala/com/nvidia/spark/rapids/delta/common/DeltaProviderBase.scala - Delta catalog
delta-lake/common/src/main/delta-40x-42x/scala/org/apache/spark/sql/delta/rapids/GpuDeltaCatalog4x.scala - DML commands
delta-lake/common/src/main/delta-40x-42x/scala/org/apache/spark/sql/delta/rapids/GpuDeleteCommand.scala - Merge commands
delta-lake/common/src/main/delta-40x-41x/scala/org/apache/spark/sql/delta/rapids/GpuMergeIntoCommand.scala - Table optimize
delta-lake/common/src/main/delta-33x-42x/scala/org/apache/spark/sql/delta/rapids/GpuOptimizeTableCommand.scala - Delta writes
delta-lake/common/src/main/delta-40x-42x/scala/org/apache/spark/sql/delta/rapids/GpuWriteIntoDeltaBase.scala - Transactions
delta-lake/common/src/main/delta-40x-42x/scala/org/apache/spark/sql/delta/rapids/GpuOptimisticTransaction.scala - Deletion vectors
delta-lake/common/src/main/delta-33x-42x/scala/org/apache/spark/sql/delta/deletionvectors/RapidsDeletionVectorStore.scala - Change feeds
delta-lake/common/src/main/delta-33x-41x/scala/com/nvidia/spark/rapids/delta/common/DeltaCDFRelationStrategy.scala - Auto compact
delta-lake/common/src/main/delta-33x-42x/scala/org/apache/spark/sql/delta/hooks/GpuAutoCompact.scala - Identity columns
delta-lake/common/src/main/delta-io/scala/org/apache/spark/sql/rapids/delta/GpuIdentityColumn.scala
Other components
- Spark jobs
- cuDF GPU
- Delta operations
Connections 24
- Spark jobs → GPU acceleration: runs accelerated jobs
- GPU acceleration → GPU manager: initializes resources
- GPU manager → cuDF GPU: initializes GPU memory
- Spark jobs → Version manager: selects manager
- Version manager → Shuffle proxy: extends proxy
- Shuffle proxy → Shuffle manager: loads and delegates
- Shuffle manager → Bounce buffers: uses buffers
- Shuffle manager → Async throttling: throttles async I/O
- Spark jobs → Partition joins: uses join compatibility
- Delta operations → Delta provider: requests table operations
- Delta provider → Delta catalog: provides catalog integration
- Delta operations → DML commands: requests delete or update
- Delta operations → Merge commands: requests merge
- Delta operations → Table optimize: requests optimize
- Delta operations → Delta writes: requests table write
- Delta writes → Transactions: uses transaction
- DML commands → Transactions: uses transaction
- Merge commands → Transactions: uses transaction
- Table optimize → Transactions: uses transaction
- DML commands → Deletion vectors: supports row deletion
- Merge commands → Deletion vectors: supports row deletion
- Delta operations → Change feeds: queries change feeds
- Delta operations → Auto compact: enables compaction
- Delta writes → Identity columns: supports identity columns