# yassinessadi/ecopass-cobol architecture

> Architecture diagram of the GitHub repository yassinessadi/ecopass-cobol, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yassinessadi/ecopass-cobol
- Repository: https://github.com/yassinessadi/ecopass-cobol

## Overview

EcoPass is a Spark DataSource V2 reader for binary record files. The main workflow is to generate a sidecar index from sequentially scanned record boundaries, then have Spark validate that index, plan partitions, and read complete records into rows. Framing supports configurable header-size and padding resolution plus unknown-status policies. The graph includes these documented capabilities and the Spark read path; some edges to unsampled indexing and Spark modules rely on SOURCE INDEX references, so uncertain wiring is omitted.

Main entry points: [`src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala`](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala) (Boundary scan), [`src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReader.scala`](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReader.scala) (Partition reader), [`src/main/scala/io/github/yassinessadi/ecopass/framing/RecordFramingPolicy.scala`](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/framing/RecordFramingPolicy.scala) (Framing policy), [`src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassDataSource.scala`](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassDataSource.scala) (Data source)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_framing["Record framing"]
  node_boundary["Record boundary"]
  node_header["Header parser"]
  node_types["Record types<br/>[RecordType.scala]"]
  node_headerresolver["Header resolver"]
  node_padding["Padding resolver"]
  node_policy["Framing policy"]
end

subgraph group_indexing["Index generation"]
  node_facade["Index facade<br/>[EcoPass.scala]"]
  node_generator["Boundary scan"]
  node_builder["Partition builder"]
  node_boundarymodel["Boundary model"]
  node_index["Index metadata<br/>[EcoPassIndex.scala]"]
  node_store[("Sidecar store")]
end

subgraph group_spark["Spark integration"]
  node_datasource["Data source"]
  node_table["Spark table<br/>[EcoPassTable.scala]"]
  node_scanbuilder["Scan builder"]
  node_scan["Scan planner<br/>[EcoPassScan.scala]"]
  node_batch["Batch planner<br/>[EcoPassBatch.scala]"]
  node_partition["Input partition"]
  node_factory["Reader factory"]
  node_reader["Partition reader"]
  node_schema["Output schema"]
end

node_user(("Spark user"))
node_binary["Binary file"]
node_rows["Spark rows"]

node_user -->|"requests index"| node_facade
node_facade -->|"delegates to"| node_generator
node_generator -->|"scans bytes"| node_binary
node_generator -->|"parses header"| node_header
node_generator -->|"resolves framing"| node_policy
node_policy -->|"resolves header"| node_headerresolver
node_policy -->|"resolves padding"| node_padding
node_header -->|"names status"| node_types
node_generator -->|"calculates extent"| node_boundary
node_boundary -->|"creates boundary"| node_boundarymodel
node_generator -->|"groups boundaries"| node_builder
node_builder -->|"builds metadata"| node_index
node_generator -->|"writes index"| node_store
node_user -->|"loads source"| node_datasource
node_datasource -->|"creates table"| node_table
node_table -->|"creates builder"| node_scanbuilder
node_scanbuilder -->|"reads sidecar"| node_store
node_scanbuilder -->|"creates scan"| node_scan
node_scan -->|"plans batch"| node_batch
node_batch -->|"creates partitions"| node_partition
node_batch -->|"creates factory"| node_factory
node_factory -->|"creates reader"| node_reader
node_reader -->|"reads range"| node_partition
node_reader -->|"parses header"| node_header
node_reader -.->|"applies framing"| node_policy
node_reader -->|"validates extent"| node_boundary
node_datasource -->|"uses schema"| node_schema
node_reader -->|"emits rows"| node_rows

click node_facade "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/core/EcoPass.scala"
click node_generator "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala"
click node_builder "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexBuilder.scala"
click node_boundary "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/framing/RecordBoundaryCalculator.scala"
click node_boundarymodel "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/indexing/RecordBoundary.scala"
click node_index "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndex.scala"
click node_store "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexStore.scala"
click node_header "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/core/RecordHeaderParser.scala"
click node_types "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/core/RecordType.scala"
click node_headerresolver "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/framing/HeaderFormatResolver.scala"
click node_padding "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/framing/PaddingResolver.scala"
click node_policy "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/framing/RecordFramingPolicy.scala"
click node_datasource "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassDataSource.scala"
click node_table "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassTable.scala"
click node_scanbuilder "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassScanBuilder.scala"
click node_scan "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassScan.scala"
click node_batch "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassBatch.scala"
click node_partition "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassInputPartition.scala"
click node_factory "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReaderFactory.scala"
click node_reader "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReader.scala"
click node_schema "https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassSchema.scala"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_boundary,node_header,node_types,node_headerresolver,node_padding,node_policy,node_user toneBlue
class node_facade,node_generator,node_builder,node_boundarymodel,node_index,node_store toneAmber
class node_datasource,node_table,node_scanbuilder,node_scan,node_batch,node_partition,node_factory,node_reader,node_schema toneMint
class node_binary,node_rows toneIndigo
```

## Components

### Record framing

- [Record boundary](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/framing/RecordBoundaryCalculator.scala): `src/main/scala/io/github/yassinessadi/ecopass/framing/RecordBoundaryCalculator.scala`
- [Header parser](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/core/RecordHeaderParser.scala): `src/main/scala/io/github/yassinessadi/ecopass/core/RecordHeaderParser.scala`
- [Record types](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/core/RecordType.scala): `src/main/scala/io/github/yassinessadi/ecopass/core/RecordType.scala`
- [Header resolver](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/framing/HeaderFormatResolver.scala): `src/main/scala/io/github/yassinessadi/ecopass/framing/HeaderFormatResolver.scala`
- [Padding resolver](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/framing/PaddingResolver.scala): `src/main/scala/io/github/yassinessadi/ecopass/framing/PaddingResolver.scala`
- [Framing policy](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/framing/RecordFramingPolicy.scala): `src/main/scala/io/github/yassinessadi/ecopass/framing/RecordFramingPolicy.scala`

### Index generation

- [Index facade](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/core/EcoPass.scala): `src/main/scala/io/github/yassinessadi/ecopass/core/EcoPass.scala`
- [Boundary scan](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala): `src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala`
- [Partition builder](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexBuilder.scala): `src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexBuilder.scala`
- [Boundary model](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/indexing/RecordBoundary.scala): `src/main/scala/io/github/yassinessadi/ecopass/indexing/RecordBoundary.scala`
- [Index metadata](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndex.scala): `src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndex.scala`
- [Sidecar store](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexStore.scala): `src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexStore.scala`

### Spark integration

- [Data source](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassDataSource.scala): `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassDataSource.scala`
- [Spark table](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassTable.scala): `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassTable.scala`
- [Scan builder](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassScanBuilder.scala): `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassScanBuilder.scala`
- [Scan planner](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassScan.scala): `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassScan.scala`
- [Batch planner](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassBatch.scala): `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassBatch.scala`
- [Input partition](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassInputPartition.scala): `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassInputPartition.scala`
- [Reader factory](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReaderFactory.scala): `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReaderFactory.scala`
- [Partition reader](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReader.scala): `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReader.scala`
- [Output schema](https://github.com/yassinessadi/ecopass-cobol/blob/main/src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassSchema.scala): `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassSchema.scala`

### Other components

- **Spark user**
- **Binary file**
- **Spark rows**

## Connections

- Spark user → Index facade: requests index (evidence: `README.md`)
- Index facade → Boundary scan: delegates to (evidence: `src/main/scala/io/github/yassinessadi/ecopass/core/EcoPass.scala`)
- Boundary scan → Binary file: scans bytes (evidence: `README.md`)
- Boundary scan → Header parser: parses header (evidence: `src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala`)
- Boundary scan → Framing policy: resolves framing (evidence: `src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala`)
- Framing policy → Header resolver: resolves header (evidence: `src/main/scala/io/github/yassinessadi/ecopass/framing/RecordFramingPolicy.scala`)
- Framing policy → Padding resolver: resolves padding (evidence: `src/main/scala/io/github/yassinessadi/ecopass/framing/RecordFramingPolicy.scala`)
- Header parser → Record types: names status (evidence: `src/main/scala/io/github/yassinessadi/ecopass/core/RecordHeaderParser.scala`)
- Boundary scan → Record boundary: calculates extent (evidence: `src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala`)
- Record boundary → Boundary model: creates boundary (evidence: `src/main/scala/io/github/yassinessadi/ecopass/framing/RecordBoundaryCalculator.scala`)
- Boundary scan → Partition builder: groups boundaries (evidence: `src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala`)
- Partition builder → Index metadata: builds metadata (evidence: `src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexBuilder.scala`)
- Boundary scan → Sidecar store: writes index (evidence: `src/main/scala/io/github/yassinessadi/ecopass/indexing/EcoPassIndexGenerator.scala`)
- Spark user → Data source: loads source (evidence: `README.md`)
- Data source → Spark table: creates table (evidence: `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassDataSource.scala`)
- Spark table → Scan builder: creates builder (evidence: `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassTable.scala`)
- Scan builder → Sidecar store: reads sidecar (evidence: `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassScanBuilder.scala`)
- Scan builder → Scan planner: creates scan (evidence: `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassScanBuilder.scala`)
- Scan planner → Batch planner: plans batch (evidence: `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassScan.scala`)
- Batch planner → Input partition: creates partitions (evidence: `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassBatch.scala`)
- Batch planner → Reader factory: creates factory (evidence: `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassBatch.scala`)
- Reader factory → Partition reader: creates reader (evidence: `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReaderFactory.scala`)
- Partition reader → Input partition: reads range (evidence: `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReader.scala`)
- Partition reader → Header parser: parses header (evidence: `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReader.scala`)
- Partition reader → Framing policy: applies framing
- Partition reader → Record boundary: validates extent (evidence: `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReader.scala`)
- Data source → Output schema: uses schema (evidence: `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassDataSource.scala`)
- Partition reader → Spark rows: emits rows (evidence: `src/main/scala/io/github/yassinessadi/ecopass/spark/EcoPassPartitionReader.scala`)
