How yassinessadi/ecopass-cobol works
EcoPass is a Spark DataSource V2 reader for binary record files. The main workflow is to generate a sidecar index from sequentially scanned record boundaries, then have Spark validate that index, plan partitions, and read complete records into rows. Framing supports configurable header-size and padding resolution plus unknown-status policies. The graph includes these documented capabilities and the Spark read path; some edges to unsampled indexing and Spark modules rely on SOURCE INDEX references, so uncertain wiring is omitted.
Main entry points: EcoPassIndexGenerator.scala Boundary scan · EcoPassPartitionReader.scala Partition reader · RecordFramingPolicy.scala Framing policy · EcoPassDataSource.scala Data source
Components 24
Record framing
- Record boundary
src/main/scala/io/github/yassinessadi/ecopass/framing/RecordBoundaryCalculator.scala - Header parser
src/main/scala/io/github/yassinessadi/ecopass/core/RecordHeaderParser.scala - Record types
src/main/scala/io/github/yassinessadi/ecopass/core/RecordType.scala - Header resolver
src/main/scala/io/github/yassinessadi/ecopass/framing/HeaderFormatResolver.scala - Padding resolver
src/main/scala/io/github/yassinessadi/ecopass/framing/PaddingResolver.scala - Framing policy
src/main/scala/io/github/yassinessadi/ecopass/framing/RecordFramingPolicy.scala
Index generation
- Index facade
src/main/scala/io/github/yassinessadi/ecopass/core/EcoPass.scala - Boundary scan
src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala - Partition builder
src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexBuilder.scala - Boundary model
src/main/scala/io/github/yassinessadi/ecopass/indexing/RecordBoundary.scala - Index metadata
src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndex.scala - Sidecar store
src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexStore.scala
Spark integration
- Data source
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassDataSource.scala - Spark table
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassTable.scala - Scan builder
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassScanBuilder.scala - Scan planner
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassScan.scala - Batch planner
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassBatch.scala - Input partition
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassInputPartition.scala - Reader factory
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReaderFactory.scala - Partition reader
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReader.scala - Output schema
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassSchema.scala
Other components
- Spark user
- Binary file
- Spark rows
Connections 28
- Spark user → Index facade: requests index
README.md - Index facade → Boundary scan: delegates to
src/main/scala/io/github/yassinessadi/ecopass/core/EcoPass.scala - Boundary scan → Binary file: scans bytes
README.md - Boundary scan → Header parser: parses header
src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala - Boundary scan → Framing policy: resolves framing
src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala - Framing policy → Header resolver: resolves header
src/main/scala/io/github/yassinessadi/ecopass/framing/RecordFramingPolicy.scala - Framing policy → Padding resolver: resolves padding
src/main/scala/io/github/yassinessadi/ecopass/framing/RecordFramingPolicy.scala - Header parser → Record types: names status
src/main/scala/io/github/yassinessadi/ecopass/core/RecordHeaderParser.scala - Boundary scan → Record boundary: calculates extent
src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala - Record boundary → Boundary model: creates boundary
src/main/scala/io/github/yassinessadi/ecopass/framing/RecordBoundaryCalculator.scala - Boundary scan → Partition builder: groups boundaries
src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala - Partition builder → Index metadata: builds metadata
src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexBuilder.scala - Boundary scan → Sidecar store: writes index
src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala - Spark user → Data source: loads source
README.md - Data source → Spark table: creates table
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassDataSource.scala - Spark table → Scan builder: creates builder
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassTable.scala - Scan builder → Sidecar store: reads sidecar
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassScanBuilder.scala - Scan builder → Scan planner: creates scan
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassScanBuilder.scala - Scan planner → Batch planner: plans batch
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassScan.scala - Batch planner → Input partition: creates partitions
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassBatch.scala - Batch planner → Reader factory: creates factory
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassBatch.scala - Reader factory → Partition reader: creates reader
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReaderFactory.scala - Partition reader → Input partition: reads range
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReader.scala - Partition reader → Header parser: parses header
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReader.scala - Partition reader → Framing policy: applies framing
- Partition reader → Record boundary: validates extent
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReader.scala - Data source → Output schema: uses schema
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassDataSource.scala - Partition reader → Spark rows: emits rows
src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReader.scala