# ayalaadaili/etl_kafka_hbase architecture

> Architecture diagram of the GitHub repository ayalaadaili/etl_kafka_hbase, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayalaadaili/etl_kafka_hbase
- Repository: https://github.com/ayalaadaili/etl_kafka_hbase

## Overview

##### Project type and purpose
- This is a real-time ETL (Extract, Transform, Load) pipeline project.
- It extracts news articles from the external NewsAPI, transforms them into a JSON schema, publishes them to a Kafka topic, then consumes them and loads them into HBase (and optionally HDFS).

##### File structure
- Top-level scripts:
- kafka_news_producer.py – the producer component
- Kafka_news_consumer.py – the main consumer writing to HBase
- hbase_consumer.py – alternate consumer that can also write to HDFS
- secret.txt – contains API keys/configuration
- No subdirectories—each script represents one architectural component.
- Reliance on Python libraries (kafka-python, happybase, newsapi-python, rich).

##### README insights
- Three main components described in “Architecture.”
- Kafka topic named “topic_aya,” HBase table “table_aya” with column family “cf1:data.”
- Optional HDFS storage if hbase_consumer.py is used.

##### Main components
a. External NewsAPI (REST service)
b. Kafka Producer (kafka_news_producer.py) on a compute node
c. Kafka Cluster (brokers and topic)
d. Kafka Consumer (Kafka_news_consumer.py) on a compute node
e. HBase Storage (table_aya, column family cf1)
f. Optional HDFS Storage (when using hbase_consumer.py)
g. Configuration store (secret.txt) – injects API keys

##### Architectural patterns and design principles
- Pipeline pattern: linear data flow from source → transform → sink.
- Separation of concerns: producer only extracts/transforms and publishes; consumer only subscribes and loads.
- External configuration (secret.txt) for credentials—avoids hard-coding keys.
- Idempotent writes: ensure consumer can safely retry writes to HBase.

##### Relevant technologies and libraries
- Producer: newsapi-python for REST calls; kafka-python for publishing.
- Consumer: kafka-python for subscribing; happybase for HBase; optional HDFS client API.
- HBase: NoSQL wide-column store; HDFS (optional) for batch storage.

- All main components included and clearly labeled.
- Topic name (“topic_aya”) and table name (“table_aya”).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Service and Configuration
    source["External NewsAPI"]:::external
    conf["secret.txt"]:::config
    doc["Architecture docs\nREADME.md"]:::doc

    %% Extraction Layer
    Producer["kafka_news_producer.py\n(newsapi-python, kafka-python)"]:::extraction

    %% Streaming Layer
    KafkaCluster["Kafka Cluster\n(topic_aya)"]:::kafka

    %% Loading Layer
    ConsumerHBase["Kafka_news_consumer.py\n(kafka-python, happybase)"]:::loading
    ConsumerAlt["hbase_consumer.py\n(kafka-python, happybase, HDFS)"]:::loading

    %% Storage
    HBase["HBase\n(table_aya, cf1:data)"]:::storage
    HDFS["HDFS (optional)"]:::storage

    %% Data Flow
    source -->|"HTTP GET articles\n(JSON)"| Producer
    conf -.->|"Load API keys"| Producer
    Producer -->|"Publish JSON to topic_aya"| KafkaCluster

    KafkaCluster -->|"Consume JSON"| ConsumerHBase
    conf -.->|"Load API keys"| ConsumerHBase
    ConsumerHBase -->|"Write rows to table_aya"| HBase

    KafkaCluster -->|"Consume JSON"| ConsumerAlt
    conf -.->|"Load API keys"| ConsumerAlt
    ConsumerAlt -->|"Write rows to table_aya"| HBase
    ConsumerAlt -.->|"Append JSON to HDFS"| HDFS

    %% Documentation Link
    doc --> source
    doc --> Producer
    doc --> KafkaCluster
    doc --> ConsumerHBase
    doc --> ConsumerAlt
    doc --> HBase
    doc --> HDFS

    %% Click Events
    click Producer "https://github.com/ayalaadaili/etl_kafka_hbase/blob/main/kafka_news_producer.py"
    click ConsumerHBase "https://github.com/ayalaadaili/etl_kafka_hbase/blob/main/Kafka_news_consumer.py"
    click ConsumerAlt "https://github.com/ayalaadaili/etl_kafka_hbase/blob/main/hbase_consumer.py"
    click conf "https://github.com/ayalaadaili/etl_kafka_hbase/blob/main/secret.txt"
    click doc "https://github.com/ayalaadaili/etl_kafka_hbase/blob/main/README.md"

    %% Styles
    classDef extraction fill:#E0F7FF,stroke:#00A0E3,color:#000
    classDef kafka fill:#E6FFEA,stroke:#0A8A0A,color:#000
    classDef loading fill:#FFF4E5,stroke:#E38A00,color:#000
    classDef storage fill:#FFE6E6,stroke:#E30A0A,color:#000
    classDef external fill:#F0F0F0,stroke:#888,color:#000
    classDef config fill:#F5E8FF,stroke:#7A27FF,color:#000
    classDef doc fill:#FFFFFF,stroke:#666,stroke-dasharray:2 2,color:#000
```
