How siddarth1423/data-engineering-platform works
This is a local, dependency-free CSV/NDJSON-to-SQLite warehouse. The principal workflow fingerprints customer and order inputs, validates and stages records, quarantines rejects, then publishes customer history and order facts transactionally. Separate CLI capabilities provide reports, reconciliation, and reproducible synthetic-data generation/benchmarking. The implementation is concentrated in pipeline.py; the graph separates its domain responsibilities. Some dispatch and reporting internals were not sampled, so those links rely on README documentation.
Main entry points: pipeline.py Pipeline CLI · generate_data.py Synthetic data generator
Components 17
Input and validation
- Run fingerprint
pipeline.py - Streaming parsers
pipeline.py - Row validation
pipeline.py - Temporary staging
pipeline.py
Warehouse publishing
- SQLite warehouse
pipeline.py - SCD2 customer dimension
pipeline.py - Date dimension
pipeline.py - Order facts
pipeline.py - Rejected records
pipeline.py - Run tracking
pipeline.py
Operations and analysis
- Pipeline CLI
pipeline.py - Analytical reports
pipeline.py - Reconciliation checks
pipeline.py - Synthetic data generator
generate_data.py
Other components
- Operator
- Customer files
- Order files
Connections 22
- Operator → Pipeline CLI: runs commands
README.md - Customer files → Pipeline CLI: supplies snapshots
README.md - Order files → Pipeline CLI: supplies events
README.md - Pipeline CLI → Run fingerprint: hashes inputs
README.md - Pipeline CLI → Streaming parsers: streams records
README.md - Streaming parsers → Row validation: parses rows
README.md - Row validation → Temporary staging: stages accepted rows
README.md - Row validation → Rejected records: records rejects
pipeline.py - Pipeline CLI → SQLite warehouse: opens session
pipeline.py - Temporary staging → SQLite warehouse: stores staged rows
README.md - Pipeline CLI → SCD2 customer dimension: publishes snapshots
README.md - Pipeline CLI → Date dimension: publishes dates
README.md - Pipeline CLI → Order facts: publishes facts
README.md - Pipeline CLI → Rejected records: publishes rejects
README.md - Pipeline CLI → Run tracking: records run metrics
README.md - Order facts → SCD2 customer dimension: resolves customer version
pipeline.py - Operator → Analytical reports: requests report
README.md - Analytical reports → SQLite warehouse: queries aggregates
README.md - Operator → Reconciliation checks: requests checks
README.md - Reconciliation checks → SQLite warehouse: checks stored data
README.md - Operator → Synthetic data generator: generates test batch
README.md - Pipeline CLI → Synthetic data generator: runs benchmarks
README.md