# ud17/kafka-crypto-data-pipeline architecture

> Architecture diagram of the GitHub repository ud17/kafka-crypto-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ud17/kafka-crypto-data-pipeline
- Repository: https://github.com/ud17/kafka-crypto-data-pipeline

## Overview

##### Main Components
- API Data Source – The external cryptocurrency API (e.g., livecoinwatch) that supplies the raw data.
- Airflow – Responsible for triggering/scheduling the Kafka producer job every 5 minutes.
- Kafka Producer – A Dockerized Python module (inside the “producer” folder) that connects to the API, parses data, and publishes to Kafka topics.
- Kafka Consumer – A Dockerized Python module (inside the “consumer” folder) that listens to Kafka topics, processes incoming data, and writes JSON files to AWS S3. Within the consumer folder, note that there are distinct files for handling writes to DynamoDB, RDS, and S3 (though the README specifies S3 interactions).
- AWS S3 – Used to store the JSON data output by the consumer.
- AWS Lambda (lambda.py at root) – Triggered by S3 events to process the new data and write it into DynamoDB.
- DynamoDB – The final data storage used for scalable querying of processed crypto data.
- AWS SNS – For error handling, where the consumer publishes detailed error messages to alert administrators.

##### Relationships and Interactions
- Airflow initiates the Kafka Producer Docker container (using the Airflow DAG found in producer/producer_dag.py) every 5 minutes.
- The Kafka Producer fetches real-time data from the API and publishes it to Kafka.
- The Kafka Consumer listens to Kafka topics, processes the data, and outputs JSON files to S3.
- Upon the insertion of new data in S3, an AWS Lambda function (implemented in lambda.py) is triggered.
- The Lambda function then processes this data and writes it into DynamoDB for persistent storage.
- Additionally, if processing fails in the consumer, error messages are forwarded via AWS SNS for alerting.

##### Important Architectural Patterns and Technologies
- Separation of Concerns – Distinct components for ingestion (producer), processing (consumer), scheduling (Airflow), and storage (S3, DynamoDB).
- Containerization – Both producer and consumer are Dockerized, making the system modular and portable.
- Event-Driven Architecture – AWS Lambda is triggered by events in S3, enabling real-time processing.
- Microservices-Style Approach – Even though it’s a pipeline, the system is organized in modular components that can be maintained and iterated independently.
- Dependency on Technologies – Apache Kafka for messaging, Docker for containerization, Airflow for orchestration, and AWS services for storage and processing.

##### Elements
– From API Data Source to Kafka Producer.
– From Producer to Kafka (the messaging system).
– From Kafka to Consumer.
– From Consumer to AWS S3.
– From S3 to Lambda (triggered by S3 events).
– From Lambda to DynamoDB.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "External Sources"
        API["API Data Source"]:::external
    end

    subgraph "Orchestration"
        AF["Airflow"]:::orchestration
    end

    subgraph "Producer Container"
        direction TB
        KP["Kafka Producer Container"]:::dockerized
        KP_Code["Producer Code"]:::dockerized
        KP_Dock["Producer Dockerfile"]:::dockerized
    end

    subgraph "Messaging"
        KB["Kafka Broker"]:::kafka
    end

    subgraph "Consumer Container"
        direction TB
        KC["Kafka Consumer Container"]:::dockerized
        KC_S3["Consumer S3"]:::dockerized
        KC_Dynamo["Consumer DynamoDB"]:::dockerized
        KC_RDS["Consumer RDS"]:::dockerized
        KC_Dock["Consumer Dockerfile"]:::dockerized
    end

    subgraph "AWS Cloud"
        S3["AWS S3"]:::aws
        L["AWS Lambda"]:::aws
        DD["DynamoDB"]:::aws
        SNS["AWS SNS"]:::aws
    end

    %% Data Flow Connections
    AF --> KP
    API --> KP
    KP --> KB
    KB --> KC
    KC --> KC_S3
    KC --> SNS
    KC_S3 --> S3
    S3 --> L
    L --> DD

    %% Click Events
    click AF "https://github.com/ud17/kafka-crypto-data-pipeline/blob/main/producer/producer_dag.py"
    click KP_Code "https://github.com/ud17/kafka-crypto-data-pipeline/blob/main/producer/main.py"
    click KP_Dock "https://github.com/ud17/kafka-crypto-data-pipeline/tree/main/producer/Dockerfile"
    click KC_S3 "https://github.com/ud17/kafka-crypto-data-pipeline/blob/main/consumer/consumer-s3.py"
    click KC_Dynamo "https://github.com/ud17/kafka-crypto-data-pipeline/blob/main/consumer/consumer-dynamodb.py"
    click KC_RDS "https://github.com/ud17/kafka-crypto-data-pipeline/blob/main/consumer/consumer-rds.py"
    click KC_Dock "https://github.com/ud17/kafka-crypto-data-pipeline/tree/main/consumer/Dockerfile"
    click L "https://github.com/ud17/kafka-crypto-data-pipeline/blob/main/lambda.py"
    click S3 "https://github.com/ud17/kafka-crypto-data-pipeline/blob/main/consumer/consumer-s3.py"
    click DD "https://github.com/ud17/kafka-crypto-data-pipeline/blob/main/consumer/consumer-dynamodb.py"

    %% Styles
    classDef external fill:#f88,stroke:#333,stroke-width:2px;
    classDef orchestration fill:#ff0,stroke:#333,stroke-width:2px;
    classDef dockerized fill:#bbf,stroke:#333,stroke-width:2px;
    classDef aws fill:#cfc,stroke:#333,stroke-width:2px;
    classDef kafka fill:#fcc,stroke:#333,stroke-width:2px;
```
