How ud17/kafka-crypto-data-pipeline works
Main Components
• API Data Source – The external cryptocurrency API (e.g., livecoinwatch) that supplies the raw data.
• Airflow – Responsible for triggering/scheduling the Kafka producer job every 5 minutes.
• Kafka Producer – A Dockerized Python module (inside the “producer” folder) that connects to the API, parses data, and publishes to Kafka topics.
• Kafka Consumer – A Dockerized Python module (inside the “consumer” folder) that listens to Kafka topics, processes incoming data, and writes JSON files to AWS S3. Within the consumer folder, note that there are distinct files for handling writes to DynamoDB, RDS, and S3 (though the README specifies S3 interactions).
• AWS S3 – Used to store the JSON data output by the consumer.
• AWS Lambda (lambda.py at root) – Triggered by S3 events to process the new data and write it into DynamoDB.
• DynamoDB – The final data storage used for scalable querying of processed crypto data.
• AWS SNS – For error handling, where the consumer publishes detailed error messages to alert administrators.
Relationships and Interactions
• Airflow initiates the Kafka Producer Docker container (using the Airflow DAG found in producer/producer_dag.py) every 5 minutes.
• The Kafka Producer fetches real-time data from the API and publishes it to Kafka.
• The Kafka Consumer listens to Kafka topics, processes the data, and outputs JSON files to S3.
• Upon the insertion of new data in S3, an AWS Lambda function (implemented in lambda.py) is triggered.
• The Lambda function then processes this data and writes it into DynamoDB for persistent storage.
• Additionally, if processing fails in the consumer, error messages are forwarded via AWS SNS for alerting.
Important Architectural Patterns and Technologies
• Separation of Concerns – Distinct components for ingestion (producer), processing (consumer), scheduling (Airflow), and storage (S3, DynamoDB).
• Containerization – Both producer and consumer are Dockerized, making the system modular and portable.
• Event-Driven Architecture – AWS Lambda is triggered by events in S3, enabling real-time processing.
• Microservices-Style Approach – Even though it’s a pipeline, the system is organized in modular components that can be maintained and iterated independently.
• Dependency on Technologies – Apache Kafka for messaging, Docker for containerization, Airflow for orchestration, and AWS services for storage and processing.
Elements
– From API Data Source to Kafka Producer.
– From Producer to Kafka (the messaging system).
– From Kafka to Consumer.
– From Consumer to AWS S3.
– From S3 to Lambda (triggered by S3 events).
– From Lambda to DynamoDB.