How erudite5297/invoice-processing-pipeline works
This repository implements a Spark-based invoice batch processor. Its principal workflow reads invoice and vendor CSVs, validates invoices, removes duplicates, enriches and filters records, aggregates payable totals, and writes result CSVs. The same batch entry point also runs SQL and RDD demonstrations. The model and utility files define a case class and a total-calculation helper, but the sampled batch code does not use them. The streaming module has no visible implementation in the supplied excerpt, so its behavior and connections are omitted.
Main entry points: BatchProcessor.scala Invoice batch processor · invoices.csv Invoice CSV · vendors.csv Vendor CSV
Components 7
Batch processing
- Invoice batch processor
src/main/scala/BatchProcessor.scala - Result CSV files
Input data
- Invoice CSV
data/input/invoices.csv - Vendor CSV
data/reference/vendors.csv
Domain support
- Invoice model
src/main/scala/Models.scala - Total helper
src/main/scala/Utils.scala
Other components
- Batch operator
Connections 4
- Batch operator → Invoice batch processor: runs
src/main/scala/BatchProcessor.scala - Invoice batch processor → Invoice CSV: reads
src/main/scala/BatchProcessor.scala - Invoice batch processor → Vendor CSV: reads
src/main/scala/BatchProcessor.scala - Invoice batch processor → Result CSV files: writes
src/main/scala/BatchProcessor.scala