# kotlin/kotlin-spark-api architecture

> Architecture diagram of the GitHub repository kotlin/kotlin-spark-api, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kotlin/kotlin-spark-api
- Repository: https://github.com/kotlin/kotlin-spark-api

## Overview

##### Project type and purpose
- This is an open-source, multi-module library providing a Kotlin API on top of Apache Spark (Scala).
- It offers Scala core extensions (type converters, wrappers), a Kotlin‐native API module, Jupyter integration, and tuple utilities.
- Users configure dependencies via Gradle/Maven, then import org.jetbrains.kotlinx.spark.api.* into their code or Jupyter notebooks.

##### File structure
- Top-level modules:
– buildSrc (centralized version/dependency definitions)
– core (Scala code extending Spark SQL and Catalyst)
– kotlin-spark-api (Kotlin API layer)
– jupyter (notebook integration: HTML rendering, magic commands)
– scala-tuples-in-kotlin (standalone tuple utilities module)
– examples (sample Spark applications & notebooks)
- Configuration/build files: Gradle Kotlin DSL (build.gradle.kts), settings.gradle.kts, gradle.properties, CI workflows.

##### README for additional insights
- Architecture is not explicitly diagrammed, but it describes layered API design:
– core Scala extensions
– Kotlin DSL and wrappers
– Jupyter notebook magic & rendering
- External dependencies: Apache Spark, Scala core, Kotlin, Jupyter kernel.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Systems
    subgraph "External Systems"
        Spark(("Apache Spark Cluster")):::external
        JupyterEnv(["Jupyter Notebook Environment"]):::doc
        Maven(("Maven Central / GitHub Packages")):::external
    end

    %% Build Infrastructure
    buildSrc(["buildSrc"]):::infra

    %% Core Module (Scala)
    subgraph "Core Module (Scala Extensions)" 
        KR["KotlinReflection.scala"]:::scala
        KW["KotlinWrappers.scala"]:::scala
        CT["CatalystTypeConverters.scala"]:::scala
        KSE["KSparkExtensions.scala"]:::scala
    end

    %% Kotlin API Module
    subgraph "kotlin-spark-api Module (Kotlin API Layer)"
        Col["Column.kt"]:::kotlin
        DS["Dataset.kt"]:::kotlin
        DSW["DataStreamWriter.kt"]:::kotlin
        SS["SparkSession.kt"]:::kotlin
        UReg["UDFRegister.kt"]:::kotlin
        UDF["UserDefinedFunction.kt"]:::kotlin
        UDFs["UserDefinedFunctions.kt"]:::kotlin
        SKV["StreamingKeyValues.kt"]:::kotlin
        ConvAPI["Conversions.kt"]:::kotlin
    end

    %% Scala Tuples Module
    subgraph "scala-tuples-in-kotlin Module (Tuple Utilities)"
        ConvT["Conversions.kt"]:::kotlin
        TB["TupleBuilders.kt"]:::kotlin
        DTB["DestructuredTupleBuilders.kt"]:::kotlin
        PE["ProductExtensions.kt"]:::kotlin
        TC["TupleConcatenation.kt"]:::kotlin
    end

    %% Jupyter Module
    subgraph "jupyter Module (Notebook Integration)"
        Intg["Integration.kt"]:::jupyter
        HTML["HtmlRendering.kt"]:::jupyter
        SI["SparkIntegration.kt"]:::jupyter
        SSI["SparkStreamingIntegration.kt"]:::jupyter
        JConf["JupyterConfiguration.kt"]:::jupyter
    end

    %% Examples Module
    subgraph "examples Module (Sample Apps & Notebooks)"
        Main["Main.kt"]:::examples
        WC["WordCount.kt"]:::examples
        UDFsEx["UDFs.kt"]:::examples
        StrEx["Streaming.kt"]:::examples
        NB1["JupyterExample.ipynb"]:::examples
        NB2["JupyterStreamingExample.ipynb"]:::examples
    end

    %% Dependencies and Flows
    buildSrc -->|provides versions| KR
    buildSrc -->|provides versions| Col
    buildSrc -->|provides versions| ConvT
    buildSrc -->|provides versions| Intg
    buildSrc -->|provides versions| Main

    KR -->|uses Spark SQL & Catalyst| Spark
    KW -->|uses Spark SQL & Catalyst| Spark
    CT -->|uses Spark SQL & Catalyst| Spark
    KSE -->|uses Spark SQL & Catalyst| Spark

    Col -->|depends on| KR
    DS -->|depends on| KR
    DSW -->|depends on| KR
    SS -->|depends on| KR
    UReg -->|depends on| KR
    UDF -->|depends on| KR
    UDFs -->|depends on| KR
    SKV -->|depends on| KR
    ConvAPI -->|depends on| KR

    Col -->|uses tuples| TB
    DS -->|uses tuples| DTB
    DSW -->|uses tuples| PE
    SS -->|uses tuples| TC

    Intg -->|depends on| Col
    HTML -->|depends on| DS
    SI -->|uses| Spark
    SSI -->|uses| Spark
    JConf -->|configures| Intg

    Main -->|depends on| Col
    WC -->|depends on| DS
    UDFsEx -->|depends on| UDFs
    StrEx -->|depends on| DSW
    NB1 -->|runs with| JupyterEnv
    NB2 -->|runs with| JupyterEnv

    %% Publishing
    KR -->|published to| Maven
    Col -->|published to| Maven
    ConvT -->|published to| Maven
    Intg -->|published to| Maven
    Main -->|published to| Maven

    %% Click Events
    click buildSrc "https://github.com/kotlin/kotlin-spark-api/tree/release/buildSrc/"
    click KR "https://github.com/kotlin/kotlin-spark-api/blob/release/core/src/main/scala/org/apache/spark/sql/KotlinReflection.scala"
    click KW "https://github.com/kotlin/kotlin-spark-api/blob/release/core/src/main/scala/org/apache/spark/sql/KotlinWrappers.scala"
    click CT "https://github.com/kotlin/kotlin-spark-api/blob/release/core/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala"
    click KSE "https://github.com/kotlin/kotlin-spark-api/blob/release/core/src/main/scala/org/jetbrains/kotlinx/spark/extensions/KSparkExtensions.scala"
    click Col "https://github.com/kotlin/kotlin-spark-api/blob/release/kotlin-spark-api/src/main/kotlin/org/jetbrains/kotlinx/spark/api/Column.kt"
    click DS "https://github.com/kotlin/kotlin-spark-api/blob/release/kotlin-spark-api/src/main/kotlin/org/jetbrains/kotlinx/spark/api/Dataset.kt"
    click DSW "https://github.com/kotlin/kotlin-spark-api/blob/release/kotlin-spark-api/src/main/kotlin/org/jetbrains/kotlinx/spark/api/DataStreamWriter.kt"
    click SS "https://github.com/kotlin/kotlin-spark-api/blob/release/kotlin-spark-api/src/main/kotlin/org/jetbrains/kotlinx/spark/api/SparkSession.kt"
    click UReg "https://github.com/kotlin/kotlin-spark-api/blob/release/kotlin-spark-api/src/main/kotlin/org/jetbrains/kotlinx/spark/api/UDFRegister.kt"
    click UDF "https://github.com/kotlin/kotlin-spark-api/blob/release/kotlin-spark-api/src/main/kotlin/org/jetbrains/kotlinx/spark/api/UserDefinedFunction.kt"
    click UDFs "https://github.com/kotlin/kotlin-spark-api/blob/release/kotlin-spark-api/src/main/kotlin/org/jetbrains/kotlinx/spark/api/UserDefinedFunctions.kt"
    click SKV "https://github.com/kotlin/kotlin-spark-api/blob/release/kotlin-spark-api/src/main/kotlin/org/jetbrains/kotlinx/spark/api/StreamingKeyValues.kt"
    click ConvAPI "https://github.com/kotlin/kotlin-spark-api/blob/release/kotlin-spark-api/src/main/kotlin/org/jetbrains/kotlinx/spark/api/Conversions.kt"
    click Intg "https://github.com/kotlin/kotlin-spark-api/blob/release/jupyter/src/main/kotlin/org/jetbrains/kotlinx/spark/api/jupyter/Integration.kt"
    click HTML "https://github.com/kotlin/kotlin-spark-api/blob/release/jupyter/src/main/kotlin/org/jetbrains/kotlinx/spark/api/jupyter/HtmlRendering.kt"
    click SI "https://github.com/kotlin/kotlin-spark-api/blob/release/jupyter/src/main/kotlin/org/jetbrains/kotlinx/spark/api/jupyter/SparkIntegration.kt"
    click SSI "https://github.com/kotlin/kotlin-spark-api/blob/release/jupyter/src/main/kotlin/org/jetbrains/kotlinx/spark/api/jupyter/SparkStreamingIntegration.kt"
    click JConf "https://github.com/kotlin/kotlin-spark-api/blob/release/jupyter/src/main/kotlin/org/jetbrains/kotlinx/spark/api/jupyter/JupyterConfiguration.kt"
    click ConvT "https://github.com/kotlin/kotlin-spark-api/blob/release/scala-tuples-in-kotlin/src/main/kotlin/org/jetbrains/kotlinx/spark/api/Conversions.kt"
    click TB "https://github.com/kotlin/kotlin-spark-api/blob/release/scala-tuples-in-kotlin/src/main/kotlin/org/jetbrains/kotlinx/spark/api/tuples/TupleBuilders.kt"
    click DTB "https://github.com/kotlin/kotlin-spark-api/blob/release/scala-tuples-in-kotlin/src/main/kotlin/org/jetbrains/kotlinx/spark/api/tuples/DestructuredTupleBuilders.kt"
    click PE "https://github.com/kotlin/kotlin-spark-api/blob/release/scala-tuples-in-kotlin/src/main/kotlin/org/jetbrains/kotlinx/spark/api/tuples/ProductExtensions.kt"
    click TC "https://github.com/kotlin/kotlin-spark-api/blob/release/scala-tuples-in-kotlin/src/main/kotlin/org/jetbrains/kotlinx/spark/api/tuples/TupleConcatenation.kt"
    click Main "https://github.com/kotlin/kotlin-spark-api/blob/release/examples/src/main/kotlin/org/jetbrains/kotlinx/spark/examples/Main.kt"
    click WC "https://github.com/kotlin/kotlin-spark-api/blob/release/examples/src/main/kotlin/org/jetbrains/kotlinx/spark/examples/WordCount.kt"
    click UDFsEx "https://github.com/kotlin/kotlin-spark-api/blob/release/examples/src/main/kotlin/org/jetbrains/kotlinx/spark/examples/UDFs.kt"
    click StrEx "https://github.com/kotlin/kotlin-spark-api/blob/release/examples/src/main/kotlin/org/jetbrains/kotlinx/spark/examples/streaming/Streaming.kt"
    click NB1 "https://github.com/kotlin/kotlin-spark-api/blob/release/examples/src/main/kotlin/org/jetbrains/kotlinx/spark/examples/JupyterExample.ipynb"
    click NB2 "https://github.com/kotlin/kotlin-spark-api/blob/release/examples/src/main/kotlin/org/jetbrains/kotlinx/spark/examples/streaming/JupyterStreamingExample.ipynb"

    %% Styles
    classDef infra fill:#DDDDDD,stroke:#999,stroke-width:1px
    classDef scala fill:#FFA500,stroke:#993300,stroke-width:1px
    classDef kotlin fill:#87CEFA,stroke:#1E90FF,stroke-width:1px
    classDef jupyter fill:#98FB98,stroke:#2E8B57,stroke-width:1px
    classDef examples fill:#DDA0DD,stroke:#993399,stroke-width:1px
    classDef external fill:#F5F5F5,stroke:#333,stroke-width:1px
    classDef doc fill:#FFFACD,stroke:#B8860B,stroke-width:1px,stroke-dasharray: 5 5
```
