How tejas004s/data-processing-analytics-pipeline works
This is a PySpark medallion lakehouse ETL and analytics pipeline for transactional CSV batches and clickstream JSON. Its main workflow is extraction and quality screening, Bronze Parquet ingestion with quarantine, Silver conformance into dimensional facts and dimensions, Gold business marts, then SQL analytics against the relational warehouse. S3 and Athena/Glue are documented serving capabilities, but the sampled orchestrator does not show their invocation; they are shown as separate optional integrations. Some subsystem links are based on README claims because their implementations were not sampled.
Main entry points: pipeline.py Pipeline orchestrator · parquet_loader.py Parquet loader · query_runner.py SQL query runner · dead_letter_queue.py Dead letter queue
Components 26
Ingestion and quality
- Pipeline orchestrator
src/pipeline.py - CSV extractor
src/extract/csv_batch_extractor.py - JSON extractor
src/extract/json_event_extractor.py - Quality rules
src/validate/data_quality_rules.py - Schema validation
src/validate/schema_validator.py - Dead letter queue
src/validate/dead_letter_queue.py - Quarantine data
data/dead_letter_queue
Lakehouse transformations
- Spark session
src/utils/spark_session.py - Bronze to Silver
src/transform/bronze_to_silver.py - Partition manager
src/transform/partition_manager.py - Silver to Gold
src/transform/silver_to_gold.py - Parquet loader
src/load/parquet_loader.py - Bronze data
data/bronze - Silver data
data/silver
Analytics and serving
- Gold data
data/gold - Warehouse loader
src/load/database_loader.py - Relational warehouse
- SQL query runner
src/analytics/query_runner.py - Business SQL analytics
sql/analytics - KPI reporter
src/analytics/kpi_reporter.py
Cloud integrations
- S3 lake client
src/aws/s3_client.py - AWS S3
- Athena manager
src/aws/athena_manager.py - Athena and Glue
Other components
- CSV and JSON sources
- Analytics consumer
Connections 29
- CSV and JSON sources → Pipeline orchestrator: supplies files
- Pipeline orchestrator → CSV extractor: extracts batches
- Pipeline orchestrator → JSON extractor: extracts events
- Pipeline orchestrator → Quality rules: validates rows
- Pipeline orchestrator → Schema validation: checks schemas
- Pipeline orchestrator → Dead letter queue: routes invalid
- Dead letter queue → Quarantine data: writes quarantine
- Pipeline orchestrator → Parquet loader: writes Bronze
- Parquet loader → Bronze data: stores Parquet
- Pipeline orchestrator → Bronze data: reads Bronze
- Pipeline orchestrator → Bronze to Silver: cleans and conforms
- Pipeline orchestrator → Partition manager: partitions data
- Pipeline orchestrator → Parquet loader: writes Silver
- Parquet loader → Silver data: stores Parquet
- Pipeline orchestrator → Silver data: reads Silver
- Pipeline orchestrator → Silver to Gold: builds marts
- Pipeline orchestrator → Parquet loader: writes Gold
- Parquet loader → Gold data: stores Parquet
- Pipeline orchestrator → Warehouse loader: loads tables
- Warehouse loader → Relational warehouse: writes tables
- Pipeline orchestrator → SQL query runner: runs analytics
- SQL query runner → Business SQL analytics: reads queries
- SQL query runner → Relational warehouse: queries tables
- Analytics consumer → Pipeline orchestrator: starts pipeline
- Pipeline orchestrator → KPI reporter: reports metrics
- S3 lake client → AWS S3: uploads lake files
- Athena manager → Athena and Glue: submits SQL
- AWS S3 → Athena and Glue: serves lake data
- Analytics consumer → Relational warehouse: consumes analytics