# officiallysingh/spring-boot-spark-kubernetes architecture

> Architecture diagram of the GitHub repository officiallysingh/spring-boot-spark-kubernetes, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/officiallysingh/spring-boot-spark-kubernetes
- Repository: https://github.com/officiallysingh/spring-boot-spark-kubernetes

## Overview

This repository combines a REST service for launching and exploring Spark task executions, reusable Spark job infrastructure, and example batch and streaming jobs. The user workflow is to submit a typed job request, have the service launch the configured Spark job, and inspect execution status; job pipelines then read, transform, and persist data. The README describes local and Kubernetes deployment, while the sampled sources show the REST-to-launcher and pipeline-to-connector relationships. Some runtime wiring and launcher details are unsampled, so those links are marked unverified or omitted.

Main entry points: [`spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java`](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java) (Log parsing stream), [`spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SparkPipelineExecutor.java`](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SparkPipelineExecutor.java) (Sales aggregation), [`spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobController.java`](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobController.java) (Job API), [`spark-job-service/src/main/java/com/ksoot/spark/launcher/SparkJobLauncher.java`](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-service/src/main/java/com/ksoot/spark/launcher/SparkJobLauncher.java) (Spark launcher)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_service["Job service"]
  node_api["Job API"]
  node_request["Launch request"]
  node_launcher["Spark launcher"]
  node_explorer["Execution explorer"]
  node_pagination["Paged results"]
  node_task_store[("Task metadata")]
end

subgraph group_commons["Spark commons"]
  node_execution_manager["Execution manager"]
  node_stream_launcher["Stream lifecycle"]
  node_connectors["Connector configuration"]
  node_mongo_connector["Mongo connector"]
  node_kafka_connector["Kafka connector"]
  node_jdbc_connector["JDBC connector<br/>[JdbcConnector.java]"]
end

subgraph group_batch["Batch sales"]
  node_sales_job["Sales job"]
  node_sales_data["Sales data setup<br/>[DataPopulator.java]"]
  node_sales_pipeline["Sales aggregation"]
  node_sales_config["Sales job settings<br/>[JobProperties.java]"]
end

subgraph group_stream["Log streaming"]
  node_log_job["Log analysis job"]
  node_log_generator["Log generator<br/>[LogsGenerator.java]"]
  node_log_pipeline["Log parsing stream"]
  node_log_config["Stream settings<br/>[JobProperties.java]"]
end

node_user(("Job operator"))
node_sales_mongo[("Sales collections")]
node_products_db[("Product catalog")]
node_kafka["Kafka topics"]
node_error_db[("Error log table")]

node_user -->|"submits job"| node_api
node_api -->|"accepts typed request"| node_request
node_api -->|"delegates launch"| node_launcher
node_launcher -.->|"launches configured job"| node_sales_job
node_launcher -.->|"launches configured job"| node_log_job
node_user -->|"queries executions"| node_explorer
node_explorer -->|"queries task history"| node_task_store
node_explorer -->|"assembles pages"| node_pagination
node_sales_job -->|"uses setup"| node_sales_data
node_sales_job -.->|"runs pipeline"| node_sales_pipeline
node_sales_pipeline -->|"reads month"| node_sales_config
node_sales_pipeline -->|"reads and writes"| node_mongo_connector
node_sales_pipeline -->|"reads products"| node_products_db
node_mongo_connector -->|"reads and writes"| node_sales_mongo
node_log_job -->|"uses generator"| node_log_generator
node_log_job -.->|"runs pipeline"| node_log_pipeline
node_log_pipeline -->|"reads settings"| node_log_config
node_log_pipeline -->|"reads options"| node_connectors
node_log_pipeline -->|"reads stream"| node_kafka_connector
node_log_pipeline -->|"writes stream"| node_jdbc_connector
node_kafka_connector -.->|"consumes topic"| node_kafka
node_jdbc_connector -.->|"persists errors"| node_error_db
node_log_pipeline -->|"starts stream"| node_stream_launcher
node_stream_launcher -->|"registers queries"| node_execution_manager

click node_api "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobController.java"
click node_request "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-service/src/main/java/com/ksoot/spark/dto/JobLaunchRequest.java"
click node_launcher "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-service/src/main/java/com/ksoot/spark/launcher/SparkJobLauncher.java"
click node_explorer "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobExplorerController.java"
click node_pagination "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-service/src/main/java/com/ksoot/spark/util/pagination/PaginatedResourceAssembler.java"
click node_execution_manager "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-commons/src/main/java/com/ksoot/spark/common/SparkExecutionManager.java"
click node_stream_launcher "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-commons/src/main/java/com/ksoot/spark/common/SparkStreamLauncher.java"
click node_connectors "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-commons/src/main/java/com/ksoot/spark/common/config/properties/ConnectorProperties.java"
click node_mongo_connector "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-commons/src/main/java/com/ksoot/spark/common/connector/MongoConnector.java"
click node_sales_job "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SalesReportJob.java"
click node_sales_data "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/DataPopulator.java"
click node_sales_pipeline "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SparkPipelineExecutor.java"
click node_sales_config "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/conf/JobProperties.java"
click node_log_job "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/LogAnalysisJob.java"
click node_log_generator "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/LogsGenerator.java"
click node_log_pipeline "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java"
click node_log_config "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/conf/JobProperties.java"
click node_kafka_connector "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-commons/src/main/java/com/ksoot/spark/common/connector/KafkaConnector.java"
click node_jdbc_connector "https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-commons/src/main/java/com/ksoot/spark/common/connector/JdbcConnector.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_request,node_launcher,node_explorer,node_pagination,node_task_store toneBlue
class node_execution_manager,node_stream_launcher,node_connectors,node_mongo_connector,node_kafka_connector,node_jdbc_connector,node_sales_mongo,node_products_db,node_error_db toneAmber
class node_sales_job,node_sales_data,node_sales_pipeline,node_sales_config toneMint
class node_log_job,node_log_generator,node_log_pipeline,node_log_config toneRose
class node_user,node_kafka toneIndigo
```

## Components

### Job service

- [Job API](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobController.java): `spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobController.java`
- [Launch request](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-service/src/main/java/com/ksoot/spark/dto/JobLaunchRequest.java): `spark-job-service/src/main/java/com/ksoot/spark/dto/JobLaunchRequest.java`
- [Spark launcher](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-service/src/main/java/com/ksoot/spark/launcher/SparkJobLauncher.java): `spark-job-service/src/main/java/com/ksoot/spark/launcher/SparkJobLauncher.java`
- [Execution explorer](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobExplorerController.java): `spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobExplorerController.java`
- [Paged results](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-service/src/main/java/com/ksoot/spark/util/pagination/PaginatedResourceAssembler.java): `spark-job-service/src/main/java/com/ksoot/spark/util/pagination/PaginatedResourceAssembler.java`
- **Task metadata**

### Spark commons

- [Execution manager](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-commons/src/main/java/com/ksoot/spark/common/SparkExecutionManager.java): `spark-job-commons/src/main/java/com/ksoot/spark/common/SparkExecutionManager.java`
- [Stream lifecycle](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-commons/src/main/java/com/ksoot/spark/common/SparkStreamLauncher.java): `spark-job-commons/src/main/java/com/ksoot/spark/common/SparkStreamLauncher.java`
- [Connector configuration](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-commons/src/main/java/com/ksoot/spark/common/config/properties/ConnectorProperties.java): `spark-job-commons/src/main/java/com/ksoot/spark/common/config/properties/ConnectorProperties.java`
- [Mongo connector](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-commons/src/main/java/com/ksoot/spark/common/connector/MongoConnector.java): `spark-job-commons/src/main/java/com/ksoot/spark/common/connector/MongoConnector.java`
- [Kafka connector](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-commons/src/main/java/com/ksoot/spark/common/connector/KafkaConnector.java): `spark-job-commons/src/main/java/com/ksoot/spark/common/connector/KafkaConnector.java`
- [JDBC connector](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-job-commons/src/main/java/com/ksoot/spark/common/connector/JdbcConnector.java): `spark-job-commons/src/main/java/com/ksoot/spark/common/connector/JdbcConnector.java`

### Batch sales

- [Sales job](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SalesReportJob.java): `spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SalesReportJob.java`
- [Sales data setup](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/DataPopulator.java): `spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/DataPopulator.java`
- [Sales aggregation](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SparkPipelineExecutor.java): `spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SparkPipelineExecutor.java`
- [Sales job settings](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/conf/JobProperties.java): `spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/conf/JobProperties.java`

### Log streaming

- [Log analysis job](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/LogAnalysisJob.java): `spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/LogAnalysisJob.java`
- [Log generator](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/LogsGenerator.java): `spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/LogsGenerator.java`
- [Log parsing stream](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java): `spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java`
- [Stream settings](https://github.com/officiallysingh/spring-boot-spark-kubernetes/blob/main/spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/conf/JobProperties.java): `spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/conf/JobProperties.java`

### Other components

- **Job operator**
- **Sales collections**
- **Product catalog**
- **Kafka topics**
- **Error log table**

## Connections

- Job operator → Job API: submits job (evidence: `README.md`)
- Job API → Launch request: accepts typed request (evidence: `spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobController.java`)
- Job API → Spark launcher: delegates launch (evidence: `spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobController.java`)
- Spark launcher → Sales job: launches configured job
- Spark launcher → Log analysis job: launches configured job
- Job operator → Execution explorer: queries executions (evidence: `spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobExplorerController.java`)
- Execution explorer → Task metadata: queries task history (evidence: `spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobExplorerController.java`)
- Execution explorer → Paged results: assembles pages (evidence: `spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobExplorerController.java`)
- Sales job → Sales data setup: uses setup (evidence: `spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SalesReportJob.java`)
- Sales job → Sales aggregation: runs pipeline
- Sales aggregation → Sales job settings: reads month (evidence: `spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SparkPipelineExecutor.java`)
- Sales aggregation → Mongo connector: reads and writes (evidence: `spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SparkPipelineExecutor.java`)
- Sales aggregation → Product catalog: reads products (evidence: `spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SparkPipelineExecutor.java`)
- Mongo connector → Sales collections: reads and writes (evidence: `spark-job-commons/src/main/java/com/ksoot/spark/common/connector/MongoConnector.java`)
- Log analysis job → Log generator: uses generator (evidence: `spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/LogAnalysisJob.java`)
- Log analysis job → Log parsing stream: runs pipeline
- Log parsing stream → Stream settings: reads settings (evidence: `spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java`)
- Log parsing stream → Connector configuration: reads options (evidence: `spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java`)
- Log parsing stream → Kafka connector: reads stream (evidence: `spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java`)
- Log parsing stream → JDBC connector: writes stream (evidence: `spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java`)
- Kafka connector → Kafka topics: consumes topic
- JDBC connector → Error log table: persists errors
- Log parsing stream → Stream lifecycle: starts stream (evidence: `spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java`)
- Stream lifecycle → Execution manager: registers queries (evidence: `spark-job-commons/src/main/java/com/ksoot/spark/common/SparkStreamLauncher.java`)
