# ren294/log-analysis-project architecture

> Architecture diagram of the GitHub repository ren294/log-analysis-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ren294/log-analysis-project
- Repository: https://github.com/ren294/log-analysis-project

## Overview

##### Project Type and Purpose
- Type: Big Data pipeline implementing the Lambda architecture

##### File Structure
- Top‐level files/folders
– codes/
- streaming.py (real‐time Spark Streaming job)
- load.py (batch Spark job to Hive)
– nifi_template/StreamProcessLog.xml (NiFi ingestion flow)
– grafana/ (preconfigured Grafana dashboard JSON)
– powerbi/ (Power BI report file and PDF)
– image/ (architecture and UI screenshots)
- Key inference: Each code artifact maps to a layer of the pipeline

##### Extract Architecture from README
- Ingestion: Apache NiFi
- Speed layer: Kafka + Spark Streaming
- Batch layer: HDFS + Spark + Hive
- Serving: Cassandra
- Analytics: Grafana (real time) + Power BI (periodic)

##### Main Components and Interactions
a. Components
1. Data Source (NASA HTTP access logs)
2. Ingestion: Apache NiFi
3. Messaging: Apache Kafka
4. Real‐time Processor: Spark Streaming
5. Batch Processor: Spark on HDFS
6. Data Warehouse: Hive (on HDFS)
7. Serving Store: Cassandra
8. Visualization: Grafana (connects to Cassandra), Power BI (connects to Hive)
b. Data Flows
– NiFi pulls raw logs → publishes to Kafka topic “nasa_log”
– Spark Streaming subscribes to Kafka → writes processed events to Cassandra and HDFS (optional checkpointing)
– Scheduled batch job (load.py) reads raw logs from HDFS → transforms and writes to Hive tables
– Hive serves Power BI via Thrift/HTTP interface
– Cassandra serves Grafana via Cassandra data source plugin
c. Architectural Pattern
– Lambda architecture: parallel real‐time and batch pipelines converge in the serving layer
– Decoupling: ingestion, processing, storage, and visualization are separate concerns
d. Key Technologies
– NiFi (XML flow template in nifi_template/)
– Kafka (distributed commit log)
– Spark Streaming & Spark Core (packages referenced in spark‐submit commands)
– Hadoop HDFS (batch storage)
– Hive (SQL‐on‐Hadoop)
– Cassandra (NoSQL store)
– Grafana/Power BI (dashboards)

##### Capturing Detail
- Break out sub‐components if needed (e.g., NiFi → ListFile, FetchFile, PublishKafka processors)
- External dependencies: Ubuntu Server, Java/Python runtimes
- Where dashboards and reports live (e.g., Grafana web UI, Power BI Desktop)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Source
    DataSource["NASA HTTP Logs"]:::external

    %% Ingestion Layer
    subgraph "Ingestion Layer"
        NiFi["Apache NiFi"]:::proc
    end

    %% Speed Layer
    subgraph "Speed Layer"
        Kafka["Apache Kafka"]:::queue
        SparkStreaming["Spark Streaming"]:::proc
    end

    %% Batch Layer
    subgraph "Batch Layer"
        HDFS["HDFS"]:::store
        SparkBatch["Spark Batch Job"]:::proc
        Hive["Hive"]:::store
    end

    %% Serving Layer
    subgraph "Serving Layer"
        Cassandra["Cassandra"]:::store
    end

    %% Analytics Layer
    subgraph "Analytics Layer"
        Grafana["Grafana Dashboard"]:::viz
        PowerBI["Power BI Report"]:::viz
        PDF["Power BI PDF Export"]:::viz
    end

    %% Data Flows
    DataSource -->|"HTTP pull"| NiFi
    NiFi -->|"Publish to kafka topic 'nasa_log'"| Kafka
    Kafka -->|"Consume from 'nasa_log'"| SparkStreaming
    SparkStreaming -->|"Write real-time events"| Cassandra
    SparkStreaming -->|"Checkpoint (optional)"| HDFS
    HDFS -->|"spark-submit load.py"| SparkBatch
    SparkBatch -->|"Write batch tables"| Hive
    Cassandra -->|"Cassandra data source"| Grafana
    Hive -->|"Thrift/HTTP interface"| PowerBI
    PowerBI -->|"Export report"| PDF

    %% Click Events
    click NiFi "https://github.com/ren294/log-analysis-project/blob/main/nifi_template/StreamProcessLog.xml"
    click SparkStreaming "https://github.com/ren294/log-analysis-project/blob/main/codes/streaming.py"
    click SparkBatch "https://github.com/ren294/log-analysis-project/blob/main/codes/load.py"
    click Grafana "https://github.com/ren294/log-analysis-project/blob/main/grafana/NasaLog-1723364517749.json"
    click PowerBI "https://github.com/ren294/log-analysis-project/blob/main/powerbi/NasaLog.pbix"
    click PDF "https://github.com/ren294/log-analysis-project/blob/main/powerbi/Report.pdf"

    %% Styles
    classDef external fill:#ffdddd,stroke:#cc0000,color:#000;
    classDef proc fill:#ddffdd,stroke:#00aa00,color:#000;
    classDef queue fill:#ddddff,stroke:#0000aa,color:#000;
    classDef store fill:#fff0b3,stroke:#b38600,color:#000;
    classDef viz fill:#ffe0cc,stroke:#cc5500,color:#000;
```
