How nayak623/supplier-data-pipeline works
This is a supplier-product integration pipeline: it reads heterogeneous supplier feeds, standardizes and validates records, resolves duplicate-product conflicts, and stores canonical products with source traceability. The documented run also exports CSV reports and records execution metrics and history. The graph follows the calls in the entry script; README statements cover the user invocation and documented capabilities. Unsampled implementations are not inferred beyond those documented behaviors.
Main entry points: run_ingestion.py Pipeline Runner · storage.py Database Storage · monitoring.py Execution Monitoring · ingestion.py Feed Ingestion
Components 12
Pipeline Workflow
- Pipeline Runner
run_ingestion.py - Supplier Feeds
data
Product Processing
- Feed Ingestion
src/ingestion.py - Schema Standardization
src/transformation.py - Record Validation
src/validation.py - Conflict Resolution
src/conflict_resolution_optimized.py
Persistence and Exports
- Database Storage
src/storage.py - Pipeline Database
output - CSV Reports
output
Run Operations
- Execution Monitoring
src/monitoring.py - Pipeline Logs
logs
Other components
- Pipeline Operator
Connections 11
- Pipeline Operator → Pipeline Runner: runs
README.md - Pipeline Runner → Feed Ingestion: loads feeds
run_ingestion.py - Supplier Feeds → Feed Ingestion: reads CSV/JSON
src/ingestion.py - Pipeline Runner → Schema Standardization: standardizes
run_ingestion.py - Pipeline Runner → Record Validation: validates
run_ingestion.py - Pipeline Runner → Conflict Resolution: resolves conflicts
run_ingestion.py - Pipeline Runner → Database Storage: stores results
run_ingestion.py - Database Storage → Pipeline Database: writes SQLite
src/storage.py - Pipeline Runner → CSV Reports: exports CSV
run_ingestion.py - Pipeline Runner → Execution Monitoring: logs metrics
run_ingestion.py - Execution Monitoring → Pipeline Logs: writes logs
src/monitoring.py