# tejas004s/data-processing-analytics-pipeline architecture

> Architecture diagram of the GitHub repository tejas004s/data-processing-analytics-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tejas004s/data-processing-analytics-pipeline
- Repository: https://github.com/tejas004s/data-processing-analytics-pipeline

## Overview

This is a PySpark medallion lakehouse ETL and analytics pipeline for transactional CSV batches and clickstream JSON. Its main workflow is extraction and quality screening, Bronze Parquet ingestion with quarantine, Silver conformance into dimensional facts and dimensions, Gold business marts, then SQL analytics against the relational warehouse. S3 and Athena/Glue are documented serving capabilities, but the sampled orchestrator does not show their invocation; they are shown as separate optional integrations. Some subsystem links are based on README claims because their implementations were not sampled.

Main entry points: [`src/pipeline.py`](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/pipeline.py) (Pipeline orchestrator), [`src/load/parquet_loader.py`](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/load/parquet_loader.py) (Parquet loader), [`src/analytics/query_runner.py`](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/analytics/query_runner.py) (SQL query runner), [`src/validate/dead_letter_queue.py`](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/validate/dead_letter_queue.py) (Dead letter queue)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion and quality"]
  node_orchestrator["Pipeline orchestrator<br/>[pipeline.py]"]
  node_csv["CSV extractor"]
  node_json["JSON extractor"]
  node_quality["Quality rules"]
  node_schema["Schema validation"]
  node_dlq["Dead letter queue"]
  node_dlq_store[("Quarantine data")]
end

subgraph group_lakehouse["Lakehouse transformations"]
  node_spark["Spark session<br/>[spark_session.py]"]
  node_bronze_transform["Bronze to Silver"]
  node_partition["Partition manager"]
  node_gold_transform["Silver to Gold<br/>[silver_to_gold.py]"]
  node_parquet["Parquet loader<br/>[parquet_loader.py]"]
  node_bronze_store[("Bronze data")]
  node_silver_store[("Silver data")]
end

subgraph group_analytics["Analytics and serving"]
  node_gold_store[("Gold data")]
  node_warehouse_loader["Warehouse loader<br/>[database_loader.py]"]
  node_warehouse[("Relational warehouse")]
  node_sql_runner["SQL query runner<br/>[query_runner.py]"]
  node_sql_analytics["Business SQL analytics"]
  node_reporter["KPI reporter<br/>[kpi_reporter.py]"]
end

subgraph group_cloud["Cloud integrations"]
  node_s3_client["S3 lake client<br/>[s3_client.py]"]
  node_s3[("AWS S3")]
  node_athena_manager["Athena manager<br/>[athena_manager.py]"]
  node_athena["Athena and Glue"]
end

node_sources(("CSV and JSON sources"))
node_analyst(("Analytics consumer"))

node_sources -->|"supplies files"| node_orchestrator
node_orchestrator -->|"extracts batches"| node_csv
node_orchestrator -->|"extracts events"| node_json
node_orchestrator -->|"validates rows"| node_quality
node_orchestrator -->|"checks schemas"| node_schema
node_orchestrator -->|"routes invalid"| node_dlq
node_dlq -->|"writes quarantine"| node_dlq_store
node_orchestrator -->|"writes Bronze"| node_parquet
node_parquet -->|"stores Parquet"| node_bronze_store
node_orchestrator -->|"reads Bronze"| node_bronze_store
node_orchestrator -->|"cleans and conforms"| node_bronze_transform
node_orchestrator -->|"partitions data"| node_partition
node_orchestrator -->|"writes Silver"| node_parquet
node_parquet -->|"stores Parquet"| node_silver_store
node_orchestrator -->|"reads Silver"| node_silver_store
node_orchestrator -->|"builds marts"| node_gold_transform
node_orchestrator -->|"writes Gold"| node_parquet
node_parquet -->|"stores Parquet"| node_gold_store
node_orchestrator -->|"loads tables"| node_warehouse_loader
node_warehouse_loader -->|"writes tables"| node_warehouse
node_orchestrator -->|"runs analytics"| node_sql_runner
node_sql_runner -->|"reads queries"| node_sql_analytics
node_sql_runner -->|"queries tables"| node_warehouse
node_analyst -->|"starts pipeline"| node_orchestrator
node_orchestrator -->|"reports metrics"| node_reporter
node_s3_client -.->|"uploads lake files"| node_s3
node_athena_manager -.->|"submits SQL"| node_athena
node_s3 -.->|"serves lake data"| node_athena
node_analyst -->|"consumes analytics"| node_warehouse

click node_orchestrator "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/pipeline.py"
click node_csv "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/extract/csv_batch_extractor.py"
click node_json "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/extract/json_event_extractor.py"
click node_quality "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/validate/data_quality_rules.py"
click node_schema "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/validate/schema_validator.py"
click node_dlq "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/validate/dead_letter_queue.py"
click node_spark "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/utils/spark_session.py"
click node_bronze_transform "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/transform/bronze_to_silver.py"
click node_partition "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/transform/partition_manager.py"
click node_gold_transform "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/transform/silver_to_gold.py"
click node_parquet "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/load/parquet_loader.py"
click node_bronze_store "https://github.com/tejas004s/data-processing-analytics-pipeline/tree/main/data/bronze"
click node_silver_store "https://github.com/tejas004s/data-processing-analytics-pipeline/tree/main/data/silver"
click node_gold_store "https://github.com/tejas004s/data-processing-analytics-pipeline/tree/main/data/gold"
click node_dlq_store "https://github.com/tejas004s/data-processing-analytics-pipeline/tree/main/data/dead_letter_queue"
click node_warehouse_loader "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/load/database_loader.py"
click node_sql_runner "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/analytics/query_runner.py"
click node_sql_analytics "https://github.com/tejas004s/data-processing-analytics-pipeline/tree/main/sql/analytics"
click node_reporter "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/analytics/kpi_reporter.py"
click node_s3_client "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/aws/s3_client.py"
click node_athena_manager "https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/aws/athena_manager.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_orchestrator,node_csv,node_json,node_quality,node_schema,node_dlq,node_dlq_store toneBlue
class node_spark,node_bronze_transform,node_partition,node_gold_transform,node_parquet,node_bronze_store,node_silver_store toneAmber
class node_gold_store,node_warehouse_loader,node_warehouse,node_sql_runner,node_sql_analytics,node_reporter toneMint
class node_s3_client,node_s3,node_athena_manager,node_athena toneRose
class node_sources,node_analyst toneIndigo
```

## Components

### Ingestion and quality

- [Pipeline orchestrator](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/pipeline.py): `src/pipeline.py`
- [CSV extractor](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/extract/csv_batch_extractor.py): `src/extract/csv_batch_extractor.py`
- [JSON extractor](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/extract/json_event_extractor.py): `src/extract/json_event_extractor.py`
- [Quality rules](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/validate/data_quality_rules.py): `src/validate/data_quality_rules.py`
- [Schema validation](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/validate/schema_validator.py): `src/validate/schema_validator.py`
- [Dead letter queue](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/validate/dead_letter_queue.py): `src/validate/dead_letter_queue.py`
- [Quarantine data](https://github.com/tejas004s/data-processing-analytics-pipeline/tree/main/data/dead_letter_queue): `data/dead_letter_queue`

### Lakehouse transformations

- [Spark session](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/utils/spark_session.py): `src/utils/spark_session.py`
- [Bronze to Silver](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/transform/bronze_to_silver.py): `src/transform/bronze_to_silver.py`
- [Partition manager](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/transform/partition_manager.py): `src/transform/partition_manager.py`
- [Silver to Gold](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/transform/silver_to_gold.py): `src/transform/silver_to_gold.py`
- [Parquet loader](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/load/parquet_loader.py): `src/load/parquet_loader.py`
- [Bronze data](https://github.com/tejas004s/data-processing-analytics-pipeline/tree/main/data/bronze): `data/bronze`
- [Silver data](https://github.com/tejas004s/data-processing-analytics-pipeline/tree/main/data/silver): `data/silver`

### Analytics and serving

- [Gold data](https://github.com/tejas004s/data-processing-analytics-pipeline/tree/main/data/gold): `data/gold`
- [Warehouse loader](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/load/database_loader.py): `src/load/database_loader.py`
- **Relational warehouse**
- [SQL query runner](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/analytics/query_runner.py): `src/analytics/query_runner.py`
- [Business SQL analytics](https://github.com/tejas004s/data-processing-analytics-pipeline/tree/main/sql/analytics): `sql/analytics`
- [KPI reporter](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/analytics/kpi_reporter.py): `src/analytics/kpi_reporter.py`

### Cloud integrations

- [S3 lake client](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/aws/s3_client.py): `src/aws/s3_client.py`
- **AWS S3**
- [Athena manager](https://github.com/tejas004s/data-processing-analytics-pipeline/blob/main/src/aws/athena_manager.py): `src/aws/athena_manager.py`
- **Athena and Glue**

### Other components

- **CSV and JSON sources**
- **Analytics consumer**

## Connections

- CSV and JSON sources → Pipeline orchestrator: supplies files
- Pipeline orchestrator → CSV extractor: extracts batches
- Pipeline orchestrator → JSON extractor: extracts events
- Pipeline orchestrator → Quality rules: validates rows
- Pipeline orchestrator → Schema validation: checks schemas
- Pipeline orchestrator → Dead letter queue: routes invalid
- Dead letter queue → Quarantine data: writes quarantine
- Pipeline orchestrator → Parquet loader: writes Bronze
- Parquet loader → Bronze data: stores Parquet
- Pipeline orchestrator → Bronze data: reads Bronze
- Pipeline orchestrator → Bronze to Silver: cleans and conforms
- Pipeline orchestrator → Partition manager: partitions data
- Pipeline orchestrator → Parquet loader: writes Silver
- Parquet loader → Silver data: stores Parquet
- Pipeline orchestrator → Silver data: reads Silver
- Pipeline orchestrator → Silver to Gold: builds marts
- Pipeline orchestrator → Parquet loader: writes Gold
- Parquet loader → Gold data: stores Parquet
- Pipeline orchestrator → Warehouse loader: loads tables
- Warehouse loader → Relational warehouse: writes tables
- Pipeline orchestrator → SQL query runner: runs analytics
- SQL query runner → Business SQL analytics: reads queries
- SQL query runner → Relational warehouse: queries tables
- Analytics consumer → Pipeline orchestrator: starts pipeline
- Pipeline orchestrator → KPI reporter: reports metrics
- S3 lake client → AWS S3: uploads lake files
- Athena manager → Athena and Glue: submits SQL
- AWS S3 → Athena and Glue: serves lake data
- Analytics consumer → Relational warehouse: consumes analytics