How ren294/log-analysis-project works
Project Type and Purpose
• Type: Big Data pipeline implementing the Lambda architecture
File Structure
• Top‐level files/folders
– codes/
• streaming.py (real‐time Spark Streaming job)
• load.py (batch Spark job to Hive)
– nifi_template/StreamProcessLog.xml (NiFi ingestion flow)
– grafana/ (preconfigured Grafana dashboard JSON)
– powerbi/ (Power BI report file and PDF)
– image/ (architecture and UI screenshots)
• Key inference: Each code artifact maps to a layer of the pipeline
Extract Architecture from README
• Ingestion: Apache NiFi
• Speed layer: Kafka + Spark Streaming
• Batch layer: HDFS + Spark + Hive
• Serving: Cassandra
• Analytics: Grafana (real time) + Power BI (periodic)
Main Components and Interactions
a. Components
1. Data Source (NASA HTTP access logs)
2. Ingestion: Apache NiFi
3. Messaging: Apache Kafka
4. Real‐time Processor: Spark Streaming
5. Batch Processor: Spark on HDFS
6. Data Warehouse: Hive (on HDFS)
7. Serving Store: Cassandra
8. Visualization: Grafana (connects to Cassandra), Power BI (connects to Hive)
b. Data Flows
– NiFi pulls raw logs → publishes to Kafka topic “nasa_log”
– Spark Streaming subscribes to Kafka → writes processed events to Cassandra and HDFS (optional checkpointing)
– Scheduled batch job (load.py) reads raw logs from HDFS → transforms and writes to Hive tables
– Hive serves Power BI via Thrift/HTTP interface
– Cassandra serves Grafana via Cassandra data source plugin
c. Architectural Pattern
– Lambda architecture: parallel real‐time and batch pipelines converge in the serving layer
– Decoupling: ingestion, processing, storage, and visualization are separate concerns
d. Key Technologies
– NiFi (XML flow template in nifi_template/)
– Kafka (distributed commit log)
– Spark Streaming & Spark Core (packages referenced in spark‐submit commands)
– Hadoop HDFS (batch storage)
– Hive (SQL‐on‐Hadoop)
– Cassandra (NoSQL store)
– Grafana/Power BI (dashboards)
Capturing Detail
• Break out sub‐components if needed (e.g., NiFi → ListFile, FetchFile, PublishKafka processors)
• External dependencies: Ubuntu Server, Java/Python runtimes
• Where dashboards and reports live (e.g., Grafana web UI, Power BI Desktop)