# aravind-mg/data-integration-platform architecture

> Architecture diagram of the GitHub repository aravind-mg/data-integration-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aravind-mg/data-integration-platform
- Repository: https://github.com/aravind-mg/data-integration-platform

## Overview

Customer360 is an airline data-integration pipeline that turns four source datasets into a customer-level profile with booking, loyalty, and support metrics. The terminal workflow generates a batch, runs it through reading, validation, cleaning, aggregation, joining, final validation, and CSV output; rejected or otherwise invalid records are also written separately. The graph preserves the documented source domains and the pipeline’s distinct processing stages. Some generator and stage internals were not sampled, so relationships rely on the README and SOURCE INDEX where applicable.

Main entry points: [`src/pipeline.py`](https://github.com/aravind-mg/data-integration-platform/blob/main/src/pipeline.py) (Pipeline orchestrator), [`data/generate_source_data.py`](https://github.com/aravind-mg/data-integration-platform/blob/main/data/generate_source_data.py) (Batch generator), [`src/config.py`](https://github.com/aravind-mg/data-integration-platform/blob/main/src/config.py) (Pipeline configuration), [`src/write_output.py`](https://github.com/aravind-mg/data-integration-platform/blob/main/src/write_output.py) (Output writer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Application entry"]
  node_main["Application entry<br/>[main.py]"]
end

subgraph group_sources["Source data"]
  node_generator["Batch generator"]
  node_customer_gen["Customer generator<br/>[customers.py]"]
  node_booking_gen["Booking generator<br/>[bookings.py]"]
  node_loyalty_gen["Loyalty generator<br/>[loyalty.py]"]
  node_support_gen["Support generator<br/>[support_tickets.py]"]
end

subgraph group_processing["Pipeline processing"]
  node_pipeline["Pipeline orchestrator<br/>[pipeline.py]"]
  node_reader["Source reader<br/>[read_data.py]"]
  node_validator["Source validation<br/>[validate_data.py]"]
  node_cleaner["Data cleaning<br/>[clean_data.py]"]
  node_aggregator["Customer metrics<br/>[aggregate_data.py]"]
  node_joiner["Customer360 join<br/>[join_data.py]"]
  node_final_validator["Final validation<br/>[final_validate.py]"]
  node_writer["Output writer<br/>[write_output.py]"]
end

subgraph group_operations["Pipeline operations"]
  node_config["Pipeline configuration<br/>[config.py]"]
  node_logger["Batch logging<br/>[logger.py]"]
end

subgraph group_artifacts["Batch artifacts"]
  node_source_batch[("Source CSV batch")]
  node_outputs[("Customer360 CSV")]
  node_invalid_output[("Invalid records CSV")]
end

node_user(("Terminal user"))

node_user -->|"runs"| node_main
node_main -->|"generates batch"| node_generator
node_main -->|"starts pipeline"| node_pipeline
node_generator -->|"generates customers"| node_customer_gen
node_generator -->|"generates bookings"| node_booking_gen
node_generator -->|"generates loyalty"| node_loyalty_gen
node_generator -->|"generates tickets"| node_support_gen
node_generator -->|"writes CSVs"| node_source_batch
node_pipeline -->|"initializes logging"| node_logger
node_pipeline -->|"reads batch"| node_reader
node_reader -->|"reads CSVs"| node_source_batch
node_pipeline -->|"validates sources"| node_validator
node_pipeline -->|"cleans records"| node_cleaner
node_pipeline -->|"aggregates metrics"| node_aggregator
node_pipeline -->|"joins customer data"| node_joiner
node_pipeline -->|"validates result"| node_final_validator
node_pipeline -->|"writes outputs"| node_writer
node_writer -->|"writes customer view"| node_outputs
node_writer -->|"writes rejected data"| node_invalid_output
node_reader -->|"uses batch settings"| node_config
node_validator -->|"uses schemas"| node_config
node_joiner -->|"uses output schema"| node_config
node_writer -->|"uses output settings"| node_config
node_logger -->|"uses log settings"| node_config

click node_main "https://github.com/aravind-mg/data-integration-platform/blob/main/main.py"
click node_generator "https://github.com/aravind-mg/data-integration-platform/blob/main/data/generate_source_data.py"
click node_customer_gen "https://github.com/aravind-mg/data-integration-platform/blob/main/data/customers.py"
click node_booking_gen "https://github.com/aravind-mg/data-integration-platform/blob/main/data/bookings.py"
click node_loyalty_gen "https://github.com/aravind-mg/data-integration-platform/blob/main/data/loyalty.py"
click node_support_gen "https://github.com/aravind-mg/data-integration-platform/blob/main/data/support_tickets.py"
click node_pipeline "https://github.com/aravind-mg/data-integration-platform/blob/main/src/pipeline.py"
click node_reader "https://github.com/aravind-mg/data-integration-platform/blob/main/src/read_data.py"
click node_validator "https://github.com/aravind-mg/data-integration-platform/blob/main/src/validate_data.py"
click node_cleaner "https://github.com/aravind-mg/data-integration-platform/blob/main/src/clean_data.py"
click node_aggregator "https://github.com/aravind-mg/data-integration-platform/blob/main/src/aggregate_data.py"
click node_joiner "https://github.com/aravind-mg/data-integration-platform/blob/main/src/join_data.py"
click node_final_validator "https://github.com/aravind-mg/data-integration-platform/blob/main/src/final_validate.py"
click node_writer "https://github.com/aravind-mg/data-integration-platform/blob/main/src/write_output.py"
click node_config "https://github.com/aravind-mg/data-integration-platform/blob/main/src/config.py"
click node_logger "https://github.com/aravind-mg/data-integration-platform/blob/main/src/logger.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_user toneBlue
class node_generator,node_customer_gen,node_booking_gen,node_loyalty_gen,node_support_gen toneAmber
class node_pipeline,node_reader,node_validator,node_cleaner,node_aggregator,node_joiner,node_final_validator,node_writer toneMint
class node_config,node_logger toneRose
class node_source_batch,node_outputs,node_invalid_output toneIndigo
```

## Components

### Application entry

- [Application entry](https://github.com/aravind-mg/data-integration-platform/blob/main/main.py): `main.py`

### Source data

- [Batch generator](https://github.com/aravind-mg/data-integration-platform/blob/main/data/generate_source_data.py): `data/generate_source_data.py`
- [Customer generator](https://github.com/aravind-mg/data-integration-platform/blob/main/data/customers.py): `data/customers.py`
- [Booking generator](https://github.com/aravind-mg/data-integration-platform/blob/main/data/bookings.py): `data/bookings.py`
- [Loyalty generator](https://github.com/aravind-mg/data-integration-platform/blob/main/data/loyalty.py): `data/loyalty.py`
- [Support generator](https://github.com/aravind-mg/data-integration-platform/blob/main/data/support_tickets.py): `data/support_tickets.py`

### Pipeline processing

- [Pipeline orchestrator](https://github.com/aravind-mg/data-integration-platform/blob/main/src/pipeline.py): `src/pipeline.py`
- [Source reader](https://github.com/aravind-mg/data-integration-platform/blob/main/src/read_data.py): `src/read_data.py`
- [Source validation](https://github.com/aravind-mg/data-integration-platform/blob/main/src/validate_data.py): `src/validate_data.py`
- [Data cleaning](https://github.com/aravind-mg/data-integration-platform/blob/main/src/clean_data.py): `src/clean_data.py`
- [Customer metrics](https://github.com/aravind-mg/data-integration-platform/blob/main/src/aggregate_data.py): `src/aggregate_data.py`
- [Customer360 join](https://github.com/aravind-mg/data-integration-platform/blob/main/src/join_data.py): `src/join_data.py`
- [Final validation](https://github.com/aravind-mg/data-integration-platform/blob/main/src/final_validate.py): `src/final_validate.py`
- [Output writer](https://github.com/aravind-mg/data-integration-platform/blob/main/src/write_output.py): `src/write_output.py`

### Pipeline operations

- [Pipeline configuration](https://github.com/aravind-mg/data-integration-platform/blob/main/src/config.py): `src/config.py`
- [Batch logging](https://github.com/aravind-mg/data-integration-platform/blob/main/src/logger.py): `src/logger.py`

### Batch artifacts

- **Source CSV batch**
- **Customer360 CSV**
- **Invalid records CSV**

### Other components

- **Terminal user**

## Connections

- Terminal user → Application entry: runs (evidence: `main.py`)
- Application entry → Batch generator: generates batch (evidence: `main.py`)
- Application entry → Pipeline orchestrator: starts pipeline (evidence: `main.py`)
- Batch generator → Customer generator: generates customers (evidence: `data/generate_source_data.py`)
- Batch generator → Booking generator: generates bookings (evidence: `data/generate_source_data.py`)
- Batch generator → Loyalty generator: generates loyalty (evidence: `data/generate_source_data.py`)
- Batch generator → Support generator: generates tickets (evidence: `data/generate_source_data.py`)
- Batch generator → Source CSV batch: writes CSVs (evidence: `data/generate_source_data.py`)
- Pipeline orchestrator → Batch logging: initializes logging (evidence: `src/pipeline.py`)
- Pipeline orchestrator → Source reader: reads batch (evidence: `src/pipeline.py`)
- Source reader → Source CSV batch: reads CSVs (evidence: `src/read_data.py`)
- Pipeline orchestrator → Source validation: validates sources (evidence: `src/pipeline.py`)
- Pipeline orchestrator → Data cleaning: cleans records (evidence: `src/pipeline.py`)
- Pipeline orchestrator → Customer metrics: aggregates metrics (evidence: `src/pipeline.py`)
- Pipeline orchestrator → Customer360 join: joins customer data (evidence: `src/pipeline.py`)
- Pipeline orchestrator → Final validation: validates result (evidence: `src/pipeline.py`)
- Pipeline orchestrator → Output writer: writes outputs (evidence: `src/pipeline.py`)
- Output writer → Customer360 CSV: writes customer view (evidence: `src/write_output.py`)
- Output writer → Invalid records CSV: writes rejected data (evidence: `src/write_output.py`)
- Source reader → Pipeline configuration: uses batch settings (evidence: `src/read_data.py`)
- Source validation → Pipeline configuration: uses schemas (evidence: `src/validate_data.py`)
- Customer360 join → Pipeline configuration: uses output schema (evidence: `src/join_data.py`)
- Output writer → Pipeline configuration: uses output settings (evidence: `src/write_output.py`)
- Batch logging → Pipeline configuration: uses log settings (evidence: `src/logger.py`)
