# siddarth1423/data-engineering-platform architecture

> Architecture diagram of the GitHub repository siddarth1423/data-engineering-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/siddarth1423/data-engineering-platform
- Repository: https://github.com/siddarth1423/data-engineering-platform

## Overview

This is a local, dependency-free CSV/NDJSON-to-SQLite warehouse. The principal workflow fingerprints customer and order inputs, validates and stages records, quarantines rejects, then publishes customer history and order facts transactionally. Separate CLI capabilities provide reports, reconciliation, and reproducible synthetic-data generation/benchmarking. The implementation is concentrated in `pipeline.py`; the graph separates its domain responsibilities. Some dispatch and reporting internals were not sampled, so those links rely on README documentation.

Main entry points: [`pipeline.py`](https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py) (Pipeline CLI), [`generate_data.py`](https://github.com/siddarth1423/data-engineering-platform/blob/main/generate_data.py) (Synthetic data generator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Input and validation"]
  node_fingerprint["Run fingerprint<br/>[pipeline.py]"]
  node_stream_parse["Streaming parsers<br/>[pipeline.py]"]
  node_validation["Row validation<br/>[pipeline.py]"]
  node_staging["Temporary staging<br/>[pipeline.py]"]
end

subgraph group_warehouse["Warehouse publishing"]
  node_warehouse_db[("SQLite warehouse<br/>[pipeline.py]")]
  node_customer_dim["SCD2 customer dimension<br/>[pipeline.py]"]
  node_date_dim["Date dimension<br/>[pipeline.py]"]
  node_order_fact["Order facts<br/>[pipeline.py]"]
  node_quarantine["Rejected records<br/>[pipeline.py]"]
  node_run_tracking["Run tracking<br/>[pipeline.py]"]
end

subgraph group_operations["Operations and analysis"]
  node_cli["Pipeline CLI<br/>[pipeline.py]"]
  node_reports["Analytical reports<br/>[pipeline.py]"]
  node_reconcile["Reconciliation checks<br/>[pipeline.py]"]
  node_generator["Synthetic data generator<br/>[generate_data.py]"]
end

node_operator(("Operator"))
node_customer_input["Customer files"]
node_order_input["Order files"]

node_operator -->|"runs commands"| node_cli
node_customer_input -->|"supplies snapshots"| node_cli
node_order_input -->|"supplies events"| node_cli
node_cli -->|"hashes inputs"| node_fingerprint
node_cli -->|"streams records"| node_stream_parse
node_stream_parse -->|"parses rows"| node_validation
node_validation -->|"stages accepted rows"| node_staging
node_validation -->|"records rejects"| node_quarantine
node_cli -->|"opens session"| node_warehouse_db
node_staging -->|"stores staged rows"| node_warehouse_db
node_cli -->|"publishes snapshots"| node_customer_dim
node_cli -->|"publishes dates"| node_date_dim
node_cli -->|"publishes facts"| node_order_fact
node_cli -->|"publishes rejects"| node_quarantine
node_cli -->|"records run metrics"| node_run_tracking
node_order_fact -->|"resolves customer version"| node_customer_dim
node_operator -->|"requests report"| node_reports
node_reports -->|"queries aggregates"| node_warehouse_db
node_operator -->|"requests checks"| node_reconcile
node_reconcile -->|"checks stored data"| node_warehouse_db
node_operator -->|"generates test batch"| node_generator
node_cli -->|"runs benchmarks"| node_generator

click node_cli "https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py"
click node_fingerprint "https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py"
click node_stream_parse "https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py"
click node_validation "https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py"
click node_staging "https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py"
click node_warehouse_db "https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py"
click node_customer_dim "https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py"
click node_date_dim "https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py"
click node_order_fact "https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py"
click node_quarantine "https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py"
click node_run_tracking "https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py"
click node_reports "https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py"
click node_reconcile "https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py"
click node_generator "https://github.com/siddarth1423/data-engineering-platform/blob/main/generate_data.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_fingerprint,node_stream_parse,node_validation,node_staging toneBlue
class node_warehouse_db,node_customer_dim,node_date_dim,node_order_fact,node_quarantine,node_run_tracking toneAmber
class node_cli,node_reports,node_reconcile,node_generator toneMint
class node_operator,node_customer_input,node_order_input toneIndigo
```

## Components

### Input and validation

- [Run fingerprint](https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py): `pipeline.py`
- [Streaming parsers](https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py): `pipeline.py`
- [Row validation](https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py): `pipeline.py`
- [Temporary staging](https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py): `pipeline.py`

### Warehouse publishing

- [SQLite warehouse](https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py): `pipeline.py`
- [SCD2 customer dimension](https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py): `pipeline.py`
- [Date dimension](https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py): `pipeline.py`
- [Order facts](https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py): `pipeline.py`
- [Rejected records](https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py): `pipeline.py`
- [Run tracking](https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py): `pipeline.py`

### Operations and analysis

- [Pipeline CLI](https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py): `pipeline.py`
- [Analytical reports](https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py): `pipeline.py`
- [Reconciliation checks](https://github.com/siddarth1423/data-engineering-platform/blob/main/pipeline.py): `pipeline.py`
- [Synthetic data generator](https://github.com/siddarth1423/data-engineering-platform/blob/main/generate_data.py): `generate_data.py`

### Other components

- **Operator**
- **Customer files**
- **Order files**

## Connections

- Operator → Pipeline CLI: runs commands (evidence: `README.md`)
- Customer files → Pipeline CLI: supplies snapshots (evidence: `README.md`)
- Order files → Pipeline CLI: supplies events (evidence: `README.md`)
- Pipeline CLI → Run fingerprint: hashes inputs (evidence: `README.md`)
- Pipeline CLI → Streaming parsers: streams records (evidence: `README.md`)
- Streaming parsers → Row validation: parses rows (evidence: `README.md`)
- Row validation → Temporary staging: stages accepted rows (evidence: `README.md`)
- Row validation → Rejected records: records rejects (evidence: `pipeline.py`)
- Pipeline CLI → SQLite warehouse: opens session (evidence: `pipeline.py`)
- Temporary staging → SQLite warehouse: stores staged rows (evidence: `README.md`)
- Pipeline CLI → SCD2 customer dimension: publishes snapshots (evidence: `README.md`)
- Pipeline CLI → Date dimension: publishes dates (evidence: `README.md`)
- Pipeline CLI → Order facts: publishes facts (evidence: `README.md`)
- Pipeline CLI → Rejected records: publishes rejects (evidence: `README.md`)
- Pipeline CLI → Run tracking: records run metrics (evidence: `README.md`)
- Order facts → SCD2 customer dimension: resolves customer version (evidence: `pipeline.py`)
- Operator → Analytical reports: requests report (evidence: `README.md`)
- Analytical reports → SQLite warehouse: queries aggregates (evidence: `README.md`)
- Operator → Reconciliation checks: requests checks (evidence: `README.md`)
- Reconciliation checks → SQLite warehouse: checks stored data (evidence: `README.md`)
- Operator → Synthetic data generator: generates test batch (evidence: `README.md`)
- Pipeline CLI → Synthetic data generator: runs benchmarks (evidence: `README.md`)