How officiallysingh/spring-boot-spark-kubernetes works
This repository combines a REST service for launching and exploring Spark task executions, reusable Spark job infrastructure, and example batch and streaming jobs. The user workflow is to submit a typed job request, have the service launch the configured Spark job, and inspect execution status; job pipelines then read, transform, and persist data. The README describes local and Kubernetes deployment, while the sampled sources show the REST-to-launcher and pipeline-to-connector relationships. Some runtime wiring and launcher details are unsampled, so those links are marked unverified or omitted.
Main entry points: loganalysis/SparkPipelineExecutor.java Log parsing stream · sales/SparkPipelineExecutor.java Sales aggregation · SparkJobController.java Job API · SparkJobLauncher.java Spark launcher
Components 25
Job service
- Job API
spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobController.java - Launch request
spark-job-service/src/main/java/com/ksoot/spark/dto/JobLaunchRequest.java - Spark launcher
spark-job-service/src/main/java/com/ksoot/spark/launcher/SparkJobLauncher.java - Execution explorer
spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobExplorerController.java - Paged results
spark-job-service/src/main/java/com/ksoot/spark/util/pagination/PaginatedResourceAssembler.java - Task metadata
Spark commons
- Execution manager
spark-job-commons/src/main/java/com/ksoot/spark/common/SparkExecutionManager.java - Stream lifecycle
spark-job-commons/src/main/java/com/ksoot/spark/common/SparkStreamLauncher.java - Connector configuration
spark-job-commons/src/main/java/com/ksoot/spark/common/config/properties/ConnectorProperties.java - Mongo connector
spark-job-commons/src/main/java/com/ksoot/spark/common/connector/MongoConnector.java - Kafka connector
spark-job-commons/src/main/java/com/ksoot/spark/common/connector/KafkaConnector.java - JDBC connector
spark-job-commons/src/main/java/com/ksoot/spark/common/connector/JdbcConnector.java
Batch sales
- Sales job
spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SalesReportJob.java - Sales data setup
spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/DataPopulator.java - Sales aggregation
spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SparkPipelineExecutor.java - Sales job settings
spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/conf/JobProperties.java
Log streaming
- Log analysis job
spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/LogAnalysisJob.java - Log generator
spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/LogsGenerator.java - Log parsing stream
spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java - Stream settings
spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/conf/JobProperties.java
Other components
- Job operator
- Sales collections
- Product catalog
- Kafka topics
- Error log table
Connections 24
- Job operator → Job API: submits job
README.md - Job API → Launch request: accepts typed request
spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobController.java - Job API → Spark launcher: delegates launch
spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobController.java - Spark launcher → Sales job: launches configured job
- Spark launcher → Log analysis job: launches configured job
- Job operator → Execution explorer: queries executions
spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobExplorerController.java - Execution explorer → Task metadata: queries task history
spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobExplorerController.java - Execution explorer → Paged results: assembles pages
spark-job-service/src/main/java/com/ksoot/spark/api/SparkJobExplorerController.java - Sales job → Sales data setup: uses setup
spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SalesReportJob.java - Sales job → Sales aggregation: runs pipeline
- Sales aggregation → Sales job settings: reads month
spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SparkPipelineExecutor.java - Sales aggregation → Mongo connector: reads and writes
spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SparkPipelineExecutor.java - Sales aggregation → Product catalog: reads products
spark-batch-sales-report-job/src/main/java/com/ksoot/spark/sales/SparkPipelineExecutor.java - Mongo connector → Sales collections: reads and writes
spark-job-commons/src/main/java/com/ksoot/spark/common/connector/MongoConnector.java - Log analysis job → Log generator: uses generator
spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/LogAnalysisJob.java - Log analysis job → Log parsing stream: runs pipeline
- Log parsing stream → Stream settings: reads settings
spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java - Log parsing stream → Connector configuration: reads options
spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java - Log parsing stream → Kafka connector: reads stream
spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java - Log parsing stream → JDBC connector: writes stream
spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java - Kafka connector → Kafka topics: consumes topic
- JDBC connector → Error log table: persists errors
- Log parsing stream → Stream lifecycle: starts stream
spark-stream-logs-analysis-job/src/main/java/com/ksoot/spark/loganalysis/SparkPipelineExecutor.java - Stream lifecycle → Execution manager: registers queries
spark-job-commons/src/main/java/com/ksoot/spark/common/SparkStreamLauncher.java