How microsoft/mobius works
1. What this repo is
Mobius is a language-binding layer that lets C# and F# code act as Spark driver applications. The repository is not a standalone app; it is a multi-language runtime bridge around Apache Spark, with .NET APIs on one side and Scala/JVM Spark components on the other.
2. Top-level architecture
The system is split into three main runtime areas:
- C#/.NET client APIs used by application code
- Scala/JVM Spark extensions running inside the Spark JVM
- A cross-process IPC layer that connects them
Supporting pieces include a C# worker process, a REPL, native socket code, build/release scripts, samples, and tests.
3. C# adapter as the public API surface
csharp/Adapter/Microsoft.Spark.CSharp is the core user-facing library. It exposes familiar Spark abstractions such as SparkContext, RDD, DataFrame, Dataset, SparkSession, SqlContext, StreamingContext, and helpers for functions, columns, rows, UDFs, and storage levels. This is the main boundary where user code enters the system.
4. Proxy-based delegation to the JVM
The adapter does not execute Spark logic itself. Most public C# objects are thin wrappers around proxy interfaces in csharp/Adapter/Microsoft.Spark.CSharp/Proxy, with concrete IPC implementations in Proxy/Ipc. These proxies forward method calls and data to the JVM side, so the C# API layer is effectively a control plane over Spark.
5. IPC bridge between .NET and Spark
Interop/Ipc contains the bridge machinery: socket communication, JSON/serialization helpers, weak object tracking, payload assembly, and JVM bridge coordination. This is the core data/control flow path for API calls, object references, and results moving between the .NET process and the Spark JVM process.
6. Networking layer and Windows-specific optimization
Network abstracts socket transport behind ISocketWrapper. There are multiple implementations:
- DefaultSocketWrapper
- SaeaSocketWrapper
- RioSocketWrapper
The cpp/Riosock native project supplies RIO-based socket support, indicating a performance-oriented Windows transport path. This layer matters because it is the low-level conduit for the bridge.
7. Spark-side Scala integration
The scala/src/main tree contains Spark extensions in Scala under packages like:
- org.apache.spark.api.csharp
- org.apache.spark.sql.api.csharp
- org.apache.spark.streaming.api.csharp
- org.apache.spark.deploy.csharp
- org.apache.spark.launcher
These classes adapt Spark internals to the C# bridge, manage backend handlers, provide C#-aware RDD/DStream behavior, and hook the runner/submission flow into Spark startup.
8. Execution model for C# functions
User-defined logic executed from Spark is handled by the C# worker side in csharp/Worker/Microsoft.Spark.CSharp. This includes Worker, TaskRunner, WorkerFunc, and multi-threaded execution support. The worker is the runtime that executes callbacks/UDFs or task logic outside the driver process, while the Scala side schedules and coordinates those tasks.
9. SQL and DataFrame subsystem
The C# Sql namespace provides the DataFrame/Dataset/Row/Column API surface and SQL utilities. On the JVM side, scala/src/main/org/apache/spark/sql/api/csharp and related Scala classes provide the Spark SQL integration. The data flow here is: C# constructs SQL/DataFrame operations, proxies them across IPC, Spark executes them in the JVM, and results/handles are returned to .NET.
10. Streaming subsystem
The C# Streaming namespace exposes StreamingContext, DStream, Kafka helpers, Event Hubs helpers, and state/window operations. Scala-side streaming integration lives under org.apache.spark.streaming.api.csharp, including C# DStream adapters and Kafka helpers. This subsystem is distinct because it bridges continuous stream processing rather than batch RDD/DataFrame operations.
11. REPL and scripting support
csharp/Repl implements a script engine and interactive shell support, backed by Roslyn. This is an alternate entry point for C# Spark code, separate from compiled apps, but it still relies on the same adapter/bridge/runtime architecture.
12. Packaging, build, and deployment flow
Build and packaging are split by language and platform:
- C# build/test scripts under csharp/
- Scala/Spark packaging under scala/
- root build/ and scripts/ for end-to-end submission/shell entrypoints
The repository also includes NuGet packaging (csharp/SparkCLR.nuspec) and Spark submission helpers (scripts/sparkclr-submit.*, scripts/sparkclr-shell.cmd).
13. Samples, examples, and validation
csharp/Samples, examples/, and scala/perf/csharp/Perf provide usage examples and performance comparisons. They are not production subsystems, but they are important architectural evidence: they exercise batch RDD, SQL/DataFrame, streaming, F#, and external sources like Kafka, Event Hubs, HDFS, Hive, Cassandra, JDBC, and Spark XML.
14. Testing boundaries
The repo has extensive tests at multiple layers:
- C# adapter unit tests in csharp/AdapterTest
- worker tests in csharp/WorkerTest
- REPL tests in csharp/ReplTest
- Scala tests in scala/src/test
- shared test helpers in csharp/Tests.Common
This reflects the architecture’s main boundaries: API wrappers, IPC/socket transport, worker execution, and Scala Spark integration.
15. Operational and environment dependencies
The README and scripts show the project targets Spark local mode and cluster deployment modes, with support for Windows and Linux. The architecture depends on Spark JVM runtime behavior, .NET runtime execution, and external systems such as HDFS, Kafka, Event Hubs, Hive, Cassandra, and JDBC when running the examples.
Main entry points: Microsoft.Spark.CSharp C# adapter · Ipc IPC core · Proxy Proxies · csharp Spark backend
Components 21
.NET side
C# and F# API surface, worker runtime, and REPL entry points.
- C# adapter · Public Spark-facing C# library exposing RDD, SQL, and streaming abstractions.
csharp/Adapter/Microsoft.Spark.CSharp - Core APIs · SparkContext, RDD, configuration, and execution primitives.
csharp/Adapter/Microsoft.Spark.CSharp/Core - SQL APIs · DataFrame, Dataset, SparkSession, Row, Column, and UDF-facing APIs.
csharp/Adapter/Microsoft.Spark.CSharp/Sql - Streaming APIs · StreamingContext, DStream, Kafka, Event Hubs, and state/window helpers.
csharp/Adapter/Microsoft.Spark.CSharp/Streaming - Proxies · Interface layer that forwards C# object behavior to IPC-backed implementations.
csharp/Adapter/Microsoft.Spark.CSharp/Proxy - IPC proxies · Concrete proxy implementations that marshal calls and results across the bridge.
csharp/Adapter/Microsoft.Spark.CSharp/Proxy/Ipc - C# worker · Separate process for executing user functions and task callbacks outside the driver.
csharp/Worker/Microsoft.Spark.CSharp - REPL · Roslyn-backed script engine and interactive entry point for Spark C# code.
csharp/Repl
IPC bridge
Cross-process transport, serialization, and object reference management.
- IPC core · Socket coordination, payload assembly, serialization, and JVM bridge state.
csharp/Adapter/Microsoft.Spark.CSharp/Interop/Ipc - Socket net · Managed socket wrappers and buffer management used by the bridge transport.
csharp/Adapter/Microsoft.Spark.CSharp/Network - RIO native · Windows RIO socket implementation backing the high-performance transport path.
cpp/Riosock
Spark JVM
Scala extensions that integrate Mobius into Spark driver and streaming execution.
- Spark backend · Spark-side backend handler and C# RDD integration in the JVM.
scala/src/main/org/apache/spark/api/csharp - SQL bridge · Spark SQL integration helpers for C# DataFrame and Dataset operations.
scala/src/main/org/apache/spark/sql/api/csharp - Streaming bridge · C# DStream adapters and streaming helpers inside the Spark JVM.
scala/src/main/org/apache/spark/streaming/api/csharp - Runner and submit · Spark submission and driver startup hooks for C# applications.
scala/src/main/org/apache/spark/deploy/csharp - C# utils · Shared Scala utility helpers used by the C# integration layers.
scala/src/main/org/apache/spark/util/csharp
Tooling
Build, scripts, samples, tests, and supporting utilities.
- Build scripts · Repo-level build, packaging, and local-mode automation.
build - Submit scripts · Shell and command wrappers for SparkCLR submission and shell startup.
scripts - Examples · Batch, SQL, streaming, and F# example applications that exercise the bridge.
examples - Tests · C# and Scala tests covering adapter, worker, REPL, bridge, and Spark integration.
csharp/AdapterTest - Docs · Project documentation and usage notes.
docs
Connections 24
- C# adapter → Core APIs: exposes · Primary public Spark API surface for driver applications.
- C# adapter → SQL APIs: exposes · Surface for SQL and DataFrame operations.
- C# adapter → Streaming APIs: exposes · Surface for streaming applications.
- Core APIs → Proxies: delegates · Core types are thin wrappers over proxy interfaces.
- SQL APIs → Proxies: delegates · SQL objects forward calls to proxy interfaces.
- Streaming APIs → Proxies: delegates · Streaming objects forward calls to proxy interfaces.
- Proxies → IPC proxies: binds to · Concrete IPC implementations realize the proxy contracts.
- IPC proxies → IPC core: uses · Proxies marshal requests through the bridge runtime.
- IPC core → Socket net: transports · Bridge messages move over managed socket wrappers.
- Socket net → RIO native: optionally uses · Windows transport can use native RIO acceleration.
- IPC core → Spark backend: communicates with · Driver-side bridge talks to Spark JVM backend handlers.
- IPC core → SQL bridge: communicates with · SQL and DataFrame calls cross the IPC boundary.
- IPC core → Streaming bridge: communicates with · Streaming control and data flows cross the IPC boundary.
- C# worker → Spark backend: executes for · Worker runtime runs user callbacks and task logic scheduled by Spark.
- Runner and submit → Spark backend: starts · Submission flow boots the C# Spark backend inside the JVM.
- Runner and submit → Submit scripts: invoked by · Shell wrappers launch SparkCLR submission and shell commands.
- REPL → C# adapter: uses · Interactive scripts still build on the same adapter surface.
- Build scripts → C# adapter: packages · Build and packaging flow produces the .NET adapter artifacts.
- Build scripts → Runner and submit: packages · Build flow also assembles Spark-side components.
- Examples → C# adapter: consumes · Sample apps demonstrate the public API surface.
- Tests → C# adapter: validates · Adapter tests verify wrapper behavior and serialization.
- Tests → IPC core: validates · Bridge tests cover socket and payload coordination.
- Tests → Spark backend: validates · Scala suites verify JVM-side integration and launch flow.
- Docs → C# adapter: documents · Documentation describes the API and usage model.