How joshua-omolewa/stock_streaming_pipeline_project works
Main Components Identification
a) Data Source Layer:
• Alpha Vantage Stock API
• Custom Stock API (FastAPI) hosted on EC2 with NGINX
• S3 Landing Zone for raw stock data
b) Data Ingestion Layer:
• Apache NiFi container
• MySQL Database container
• Debezium CDC container
c) Stream Processing Layer:
• Amazon MSK (Kafka)
• Amazon EMR with Spark Streaming
• Apache Livy
d) Orchestration Layer:
• Apache Airflow
• AWS Lambda Functions
e) Storage & Processing Layer:
• S3 (transformed data with HUDI format)
• AWS Glue
• Amazon Athena
f) Visualization Layer:
• Power BI
• Tableau
Connection Flows
• Data flow paths
• Control flow paths (different line style)
• Latency or frequency indicators where applicable
Key Processes to Highlight
a) Ingestion Flow:
Stock API → NiFi → MySQL → Debezium → MSK
b) Processing Flow:
MSK → Spark Streaming → S3 (HUDI) → Glue → Athena
c) Orchestration Flow:
API → Airflow → Lambda → Livy → EMR
Additional Elements
• A monitoring section showing Great Expectations data quality checks
• Timing indicators for real-time aspects (30-second updates)
• Container boundaries for dockerized components
Technical Details
• EC2 instance groupings
• Docker container relationships
• MSK topic structure
• Security groups and network boundaries
• Data format transformations (CSV → HUDI)