# dlimbachiya7/ifta-aws-pipeline architecture

> Architecture diagram of the GitHub repository dlimbachiya7/ifta-aws-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dlimbachiya7/ifta-aws-pipeline
- Repository: https://github.com/dlimbachiya7/ifta-aws-pipeline

## Overview

1. **Overall shape**
This repo implements an AWS medallion-style data pipeline for IFTA fuel-tax evidence. The core boundary is between **immutable raw evidence** in S3 and **curated analytical tables** in downstream S3-backed Glue/Athena datasets.

2. **Infrastructure layer**
`terraform/` provisions the cloud foundation: S3 buckets, KMS encryption, IAM roles/policies, Glue Data Catalog resources, and Athena-related setup. This is the deployment boundary for the whole system; the application logic assumes those resources already exist.

3. **Raw zone**
The raw zone is the legal/audit record of submitted evidence. It stores original files as-is in S3 and is treated as immutable/version-protected. Inputs include heterogeneous source formats: Excel trip logs, scanned PDFs for distance logs, and fuel receipts.

4. **Bronze extraction subsystem**
`glue_jobs/glue_extract.py` is the main code artifact in the repo. It runs as an **AWS Glue Python Shell** job and converts raw evidence into typed Parquet records in the bronze layer, while preserving source lineage and extraction metadata on every row.

5. **Extraction technologies and source-specific handling**
The extraction job branches by file type:
- **Textract `start_expense_analysis`** for receipt PDFs
- **Textract `start_document_analysis` with TABLES** for scanned log PDFs
- **pandas `read_excel`** for Excel trip logs  
This is the primary transformation boundary where unstructured or semi-structured evidence becomes tabular data.

6. **Bronze schema contract**
Bronze records carry both business fields and lineage fields such as `src_uri`, `src_sha256`, `src_page_or_row`, `extraction_method`, `extraction_confidence`, `ingested_at`, and `processed_at`. This makes bronze the canonical typed extraction layer and the handoff point for all downstream cleaning.

7. **Silver and gold transformation subsystem**
A separate **Glue Studio Visual ETL** job consumes bronze and produces both silver and gold outputs. Silver normalizes records into curated entities like `silver_juris_km` and `silver_fuel`, while gold aggregates by IFTA period and jurisdiction into `gold_ifta_reconciliation`.

8. **Business reconciliation logic**
The gold layer is the audit-facing analytics boundary. It performs a full outer join between kilometers and fuel totals to expose mismatches, then derives audit flags such as “no fuel in jurisdiction” or “fuel but no travel.” This layer is optimized for filing/reconciliation, not raw analysis.

9. **Catalog and query access**
All curated datasets are registered in the **AWS Glue Data Catalog** under the `ifta_medallion` database and queried through **Athena**. The catalog is the metadata boundary that makes bronze/silver/gold discoverable and queryable without embedding storage details into consumers.

10. **Data quality and lineage model**
`docs/data-quality-codes.md` and `docs/data-model.md` describe a design choice: quality issues are recorded explicitly as data rather than silently corrected. Combined with the lineage columns, this allows an auditor to trace any gold record back to its source byte and inspect why a record was accepted, corrected, or flagged.

11. **Operational flow**
The intended runtime sequence is: deploy infra with Terraform, upload the Glue extraction script, land files in raw S3, run the extraction job to populate bronze, run a crawler to register tables, run the Glue Studio ETL to produce silver/gold, then query results in Athena. Each stage can be rerun independently, with recovery costs increasing as you move from raw to gold.

12. **Repository structure as architecture**
The repo is intentionally split by concern: `terraform/` for cloud provisioning, `glue_jobs/` for executable extraction logic, and `docs/` for schema/rationale/quality rules. That separation mirrors the runtime architecture: infrastructure, ingestion, transformation, and governance.

Main entry points: [`glue_jobs/glue_extract.py`](https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/glue_jobs/glue_extract.py) (Extract job), [`docs/data-model.md`](https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/docs/data-model.md) (Data model), [`terraform/main.tf`](https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/terraform/main.tf) (Terraform core), [`docs/architecture.md`](https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/docs/architecture.md) (Architecture)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_infrastructure["Infrastructure"]
  node_terraform_main["Terraform core<br/>IaC<br/>[main.tf]"]
  node_terraform_vars["Variables<br/>IaC inputs<br/>[variables.tf]"]
  node_terraform_outputs["Outputs<br/>[outputs.tf]"]
  node_terraform_tfvars["Example vars<br/>sample config<br/>[example.tfvars]"]
end

subgraph group_ingestion["Raw and Bronze"]
  node_raw_s3[("Raw S3<br/>immutable landing")]
  node_glue_extract["Extract job<br/>Glue Python Shell<br/>[glue_extract.py]"]
  node_receipt_parse{{"Receipt OCR<br/>Textract expense<br/>[glue_extract.py]"}}
  node_log_parse{{"Log OCR<br/>Textract tables<br/>[glue_extract.py]"}}
  node_excel_parse{{"Excel ingest<br/>pandas excel<br/>[glue_extract.py]"}}
  node_bronze_parquet[("Bronze Parquet<br/>typed extraction")]
end

subgraph group_curation["Silver and Gold"]
  node_glue_crawler["Crawler"]
  node_glue_studio["Studio ETL<br/>Visual ETL"]
  node_silver_juris_km[("Juris km<br/>silver table")]
  node_silver_fuel[("Fuel table<br/>silver table")]
  node_gold_ifta[("IFTA recon<br/>gold table")]
  node_glue_catalog[("Glue Catalog<br/>metadata catalog")]
  node_athena["Athena<br/>query engine"]
end

subgraph group_governance["Governance and Docs"]
  node_data_model["Data model<br/>schema contract<br/>[data-model.md]"]
  node_quality_codes["Quality codes<br/>quality contract"]
  node_architecture_doc["Architecture<br/>design doc<br/>[architecture.md]"]
end

node_terraform_main -->|"provisions"| node_raw_s3
node_terraform_main -->|"provisions"| node_glue_catalog
node_terraform_main -->|"provisions"| node_athena
node_terraform_vars -->|"configures"| node_terraform_main
node_terraform_tfvars -->|"example"| node_terraform_vars
node_terraform_outputs -->|"references"| node_glue_extract
node_raw_s3 -->|"inputs"| node_glue_extract
node_glue_extract -->|"branches"| node_receipt_parse
node_glue_extract -->|"branches"| node_log_parse
node_glue_extract -->|"branches"| node_excel_parse
node_receipt_parse -->|"writes"| node_bronze_parquet
node_log_parse -->|"writes"| node_bronze_parquet
node_excel_parse -->|"writes"| node_bronze_parquet
node_bronze_parquet -->|"catalogs"| node_glue_crawler
node_glue_crawler -->|"registers"| node_glue_catalog
node_glue_catalog -->|"reads metadata"| node_glue_studio
node_bronze_parquet -->|"inputs"| node_glue_studio
node_glue_studio -->|"produces"| node_silver_juris_km
node_glue_studio -->|"produces"| node_silver_fuel
node_silver_juris_km -->|"joins"| node_gold_ifta
node_silver_fuel -->|"joins"| node_gold_ifta
node_gold_ifta -->|"registered in"| node_glue_catalog
node_glue_catalog -->|"queries"| node_athena
node_data_model -.->|"defines"| node_bronze_parquet
node_data_model -.->|"defines"| node_silver_juris_km
node_data_model -.->|"defines"| node_silver_fuel
node_data_model -.->|"defines"| node_gold_ifta
node_quality_codes -.->|"governs"| node_gold_ifta
node_architecture_doc -.->|"describes"| node_glue_extract
node_architecture_doc -.->|"describes"| node_glue_studio

click node_terraform_main "https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/terraform/main.tf"
click node_terraform_vars "https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/terraform/variables.tf"
click node_terraform_outputs "https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/terraform/outputs.tf"
click node_terraform_tfvars "https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/terraform/example.tfvars"
click node_glue_extract "https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/glue_jobs/glue_extract.py"
click node_receipt_parse "https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/glue_jobs/glue_extract.py"
click node_log_parse "https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/glue_jobs/glue_extract.py"
click node_excel_parse "https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/glue_jobs/glue_extract.py"
click node_data_model "https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/docs/data-model.md"
click node_quality_codes "https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/docs/data-quality-codes.md"
click node_architecture_doc "https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/docs/architecture.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_terraform_main,node_terraform_vars,node_terraform_outputs,node_terraform_tfvars toneBlue
class node_raw_s3,node_glue_extract,node_receipt_parse,node_log_parse,node_excel_parse,node_bronze_parquet toneAmber
class node_glue_crawler,node_glue_studio,node_silver_juris_km,node_silver_fuel,node_gold_ifta,node_glue_catalog,node_athena toneMint
class node_data_model,node_quality_codes,node_architecture_doc toneRose
```

## Components

### Infrastructure

Cloud foundation and deployment boundary

- [Terraform core](https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/terraform/main.tf): IaC · Provisions the AWS data pipeline foundation and shared resources · `terraform/main.tf`
- [Variables](https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/terraform/variables.tf): IaC inputs · Configures deploy-time parameters for the infrastructure stack · `terraform/variables.tf`
- [Outputs](https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/terraform/outputs.tf): IaC outputs · Exposes deployed resource references for downstream use · `terraform/outputs.tf`
- [Example vars](https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/terraform/example.tfvars): sample config · Example values for deploying the infrastructure stack · `terraform/example.tfvars`

### Raw and Bronze

Immutable evidence landing and typed extraction

- **Raw S3**: immutable landing · Version-protected original evidence stored as submitted
- [Extract job](https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/glue_jobs/glue_extract.py): Glue Python Shell · Converts raw evidence into typed bronze Parquet with lineage · `glue_jobs/glue_extract.py`
- [Receipt OCR](https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/glue_jobs/glue_extract.py): Textract expense · Extracts receipt fields from scanned fuel receipt PDFs · `glue_jobs/glue_extract.py`
- [Log OCR](https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/glue_jobs/glue_extract.py): Textract tables · Extracts trip and distance tables from scanned log PDFs · `glue_jobs/glue_extract.py`
- [Excel ingest](https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/glue_jobs/glue_extract.py): pandas excel · Reads structured trip logs from spreadsheet sources · `glue_jobs/glue_extract.py`
- **Bronze Parquet**: typed extraction · Canonical extracted records with source lineage and metadata

### Silver and Gold

Curated transformation and reconciliation outputs

- **Crawler**: Glue crawler · Registers curated files into the catalog for query and ETL
- **Studio ETL**: Visual ETL · Consumes bronze and produces silver and gold datasets
- **Juris km**: silver table · Normalizes distance by jurisdiction and period
- **Fuel table**: silver table · Normalizes fuel transactions into curated facts
- **IFTA recon**: gold table · Period and jurisdiction reconciliation with audit flags
- **Glue Catalog**: metadata catalog · Registers ifta_medallion datasets for discovery
- **Athena**: query engine · Queries curated tables through the Glue catalog

### Governance and Docs

Schema, quality, and architecture references

- [Data model](https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/docs/data-model.md): schema contract · Defines curated entities, lineage columns, and table shapes · `docs/data-model.md`
- [Quality codes](https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/docs/data-quality-codes.md): quality contract · Documents explicit data-quality states and audit semantics · `docs/data-quality-codes.md`
- [Architecture](https://github.com/dlimbachiya7/ifta-aws-pipeline/blob/main/docs/architecture.md): design doc · Explains the medallion pipeline and runtime flow · `docs/architecture.md`

## Connections

- Terraform core → Raw S3: provisions · Creates the S3 landing and supporting cloud resources
- Terraform core → Glue Catalog: provisions · Creates Glue Data Catalog resources and database
- Terraform core → Athena: provisions · Sets up query access dependencies
- Variables → Terraform core: configures · Supplies deploy-time inputs
- Example vars → Variables: example · Provides sample variable values
- Outputs → Extract job: references · Exposes deployed endpoints and resource names for jobs
- Raw S3 → Extract job: inputs · Raw evidence is read by the extraction job
- Extract job → Receipt OCR: branches · Routes receipt PDFs to Textract expense analysis
- Extract job → Log OCR: branches · Routes scanned logs to Textract document analysis
- Extract job → Excel ingest: branches · Routes spreadsheets to pandas-based parsing
- Receipt OCR → Bronze Parquet: writes · Emits typed bronze receipt records
- Log OCR → Bronze Parquet: writes · Emits typed bronze log records
- Excel ingest → Bronze Parquet: writes · Emits typed bronze trip records
- Bronze Parquet → Crawler: catalogs · Crawler discovers bronze and curated Parquet objects
- Crawler → Glue Catalog: registers · Publishes table metadata into the catalog
- Glue Catalog → Studio ETL: reads metadata · Visual ETL uses cataloged bronze tables as inputs
- Bronze Parquet → Studio ETL: inputs · Bronze records are transformed into curated outputs
- Studio ETL → Juris km: produces · Writes normalized jurisdiction-kilometer facts
- Studio ETL → Fuel table: produces · Writes normalized fuel facts
- Juris km → IFTA recon: joins · Contributes distance totals to reconciliation
- Fuel table → IFTA recon: joins · Contributes fuel totals to reconciliation
- IFTA recon → Glue Catalog: registered in · Gold tables are made discoverable in the catalog
- Glue Catalog → Athena: queries · Athena reads the registered datasets
- Data model → Bronze Parquet: defines · Specifies lineage fields and bronze contract
- Data model → Juris km: defines · Specifies curated entity structure
- Data model → Fuel table: defines · Specifies curated entity structure
- Data model → IFTA recon: defines · Specifies reconciliation outputs and audit fields
- Quality codes → IFTA recon: governs · Defines explicit quality statuses and audit semantics
- Architecture → Extract job: describes · Documents the extraction boundary and runtime flow
- Architecture → Studio ETL: describes · Documents the silver/gold transformation stage