How aravind-mg/data-integration-platform works
Customer360 is an airline data-integration pipeline that turns four source datasets into a customer-level profile with booking, loyalty, and support metrics. The terminal workflow generates a batch, runs it through reading, validation, cleaning, aggregation, joining, final validation, and CSV output; rejected or otherwise invalid records are also written separately. The graph preserves the documented source domains and the pipeline’s distinct processing stages. Some generator and stage internals were not sampled, so relationships rely on the README and SOURCE INDEX where applicable.
Main entry points: pipeline.py Pipeline orchestrator · generate_source_data.py Batch generator · config.py Pipeline configuration · write_output.py Output writer
Components 20
Application entry
- Application entry
main.py
Source data
- Batch generator
data/generate_source_data.py - Customer generator
data/customers.py - Booking generator
data/bookings.py - Loyalty generator
data/loyalty.py - Support generator
data/support_tickets.py
Pipeline processing
- Pipeline orchestrator
src/pipeline.py - Source reader
src/read_data.py - Source validation
src/validate_data.py - Data cleaning
src/clean_data.py - Customer metrics
src/aggregate_data.py - Customer360 join
src/join_data.py - Final validation
src/final_validate.py - Output writer
src/write_output.py
Pipeline operations
- Pipeline configuration
src/config.py - Batch logging
src/logger.py
Batch artifacts
- Source CSV batch
- Customer360 CSV
- Invalid records CSV
Other components
- Terminal user
Connections 24
- Terminal user → Application entry: runs
main.py - Application entry → Batch generator: generates batch
main.py - Application entry → Pipeline orchestrator: starts pipeline
main.py - Batch generator → Customer generator: generates customers
data/generate_source_data.py - Batch generator → Booking generator: generates bookings
data/generate_source_data.py - Batch generator → Loyalty generator: generates loyalty
data/generate_source_data.py - Batch generator → Support generator: generates tickets
data/generate_source_data.py - Batch generator → Source CSV batch: writes CSVs
data/generate_source_data.py - Pipeline orchestrator → Batch logging: initializes logging
src/pipeline.py - Pipeline orchestrator → Source reader: reads batch
src/pipeline.py - Source reader → Source CSV batch: reads CSVs
src/read_data.py - Pipeline orchestrator → Source validation: validates sources
src/pipeline.py - Pipeline orchestrator → Data cleaning: cleans records
src/pipeline.py - Pipeline orchestrator → Customer metrics: aggregates metrics
src/pipeline.py - Pipeline orchestrator → Customer360 join: joins customer data
src/pipeline.py - Pipeline orchestrator → Final validation: validates result
src/pipeline.py - Pipeline orchestrator → Output writer: writes outputs
src/pipeline.py - Output writer → Customer360 CSV: writes customer view
src/write_output.py - Output writer → Invalid records CSV: writes rejected data
src/write_output.py - Source reader → Pipeline configuration: uses batch settings
src/read_data.py - Source validation → Pipeline configuration: uses schemas
src/validate_data.py - Customer360 join → Pipeline configuration: uses output schema
src/join_data.py - Output writer → Pipeline configuration: uses output settings
src/write_output.py - Batch logging → Pipeline configuration: uses log settings
src/logger.py