How ayalaadaili/etl_kafka_hbase works
Project type and purpose
• This is a real-time ETL (Extract, Transform, Load) pipeline project.
• It extracts news articles from the external NewsAPI, transforms them into a JSON schema, publishes them to a Kafka topic, then consumes them and loads them into HBase (and optionally HDFS).
File structure
• Top-level scripts:
• kafka_news_producer.py – the producer component
• Kafka_news_consumer.py – the main consumer writing to HBase
• hbase_consumer.py – alternate consumer that can also write to HDFS
• secret.txt – contains API keys/configuration
• No subdirectories—each script represents one architectural component.
• Reliance on Python libraries (kafka-python, happybase, newsapi-python, rich).
README insights
• Three main components described in “Architecture.”
• Kafka topic named “topic_aya,” HBase table “table_aya” with column family “cf1:data.”
• Optional HDFS storage if hbase_consumer.py is used.
Main components
a. External NewsAPI (REST service)
b. Kafka Producer (kafka_news_producer.py) on a compute node
c. Kafka Cluster (brokers and topic)
d. Kafka Consumer (Kafka_news_consumer.py) on a compute node
e. HBase Storage (table_aya, column family cf1)
f. Optional HDFS Storage (when using hbase_consumer.py)
g. Configuration store (secret.txt) – injects API keys
Architectural patterns and design principles
• Pipeline pattern: linear data flow from source → transform → sink.
• Separation of concerns: producer only extracts/transforms and publishes; consumer only subscribes and loads.
• External configuration (secret.txt) for credentials—avoids hard-coding keys.
• Idempotent writes: ensure consumer can safely retry writes to HBase.
Relevant technologies and libraries
• Producer: newsapi-python for REST calls; kafka-python for publishing.
• Consumer: kafka-python for subscribing; happybase for HBase; optional HDFS client API.
• HBase: NoSQL wide-column store; HDFS (optional) for batch storage.
• All main components included and clearly labeled.
• Topic name (“topic_aya”) and table name (“table_aya”).