How kotlin/kotlin-spark-api works
Project type and purpose
• This is an open-source, multi-module library providing a Kotlin API on top of Apache Spark (Scala).
• It offers Scala core extensions (type converters, wrappers), a Kotlin‐native API module, Jupyter integration, and tuple utilities.
• Users configure dependencies via Gradle/Maven, then import org.jetbrains.kotlinx.spark.api.* into their code or Jupyter notebooks.
File structure
• Top-level modules:
– buildSrc (centralized version/dependency definitions)
– core (Scala code extending Spark SQL and Catalyst)
– kotlin-spark-api (Kotlin API layer)
– jupyter (notebook integration: HTML rendering, magic commands)
– scala-tuples-in-kotlin (standalone tuple utilities module)
– examples (sample Spark applications & notebooks)
• Configuration/build files: Gradle Kotlin DSL (build.gradle.kts), settings.gradle.kts, gradle.properties, CI workflows.
README for additional insights
• Architecture is not explicitly diagrammed, but it describes layered API design:
– core Scala extensions
– Kotlin DSL and wrappers
– Jupyter notebook magic & rendering
• External dependencies: Apache Spark, Scala core, Kotlin, Jupyter kernel.