How luke-who/pinterest-data-pipeline542 works
Main Components to Highlight
• Data Ingestion:
• MySQL database (either local or Amazon RDS)
• Extraction of different data types (Pinterest post, geolocation, user data)
• Batch Processing:
• Kafka REST Proxy hosted on an EC2 instance (with user_posting_emulation.py as the core script)
• Kafka Connect for moving data into S3
• Streaming Processing:
• Amazon Kinesis (managed by user_posting_emulation_streaming.py)
• Data Transformation & Analysis:
• Databricks, which uses notebooks for transformation (e.g., pinterest_data_pipeline.ipynb)
• Orchestration:
• Apache Airflow with its DAGs (as evident from the airflow/dags directory)
• Infrastructure & External Dependencies:
• Additional AWS services: API Gateway, S3 for the data lake, MWAA for managed Airflow, and AWS Cloud Services in general
Relationships and Data Flow
• Data is initialy extracted from the MySQL database.
• For batch processing, data flows from the database into Kafka via the REST proxy (running on EC2), which is then connected through Kafka Connect to S3 storage. Databricks then accesses and transforms the data.
• For real-time processing, the data is piped into Amazon Kinesis. Kinesis streams then directly integrate with Databricks for immediate processing.
• Airflow orchestrates the entire pipeline, triggering jobs, managing dependencies between batch and stream processing, and integrating with Databricks for transformation steps.
Important Architectural Patterns and Design Principles
• Separation of Concerns:
• Clearly separate ingestion, processing, transformation, and orchestration layers.
• Scalability & Fault Tolerance:
• Use managed services (like MWAA for Airflow, Kinesis for streaming) that provide scalability.
• Hybrid Setup:
• Modularity:
• Core scripts (Python scripts and Jupyter notebooks) handle specific tasks, making it easier to update or extend individual components.
Relevant Technologies and Frameworks
• Python (as the central programming language)
• MySQL (for the database)
• Apache Kafka (batch ingestion)
• Amazon Kinesis (streaming data)
• Apache Airflow (workflow orchestration)
• Databricks (data transformation and analysis)
• AWS services like S3, EC2, API Gateway, and MWAA
Detailed Elements
• Clear component labeling for every piece of the system.
• Data Sources
• Processing Layers (batch vs. streaming)
• Orchestration Modules
• External/Managed Services
• If possible, include a mini-overview of the hybrid architecture showing both local (e.g., local Apache Airflow, local MySQL) and cloud-based components (AWS services and Databricks).
Final Advice
• Maintain simplicity in visuals while ensuring all critical components and interactions are represented.