# salveendutt/big-data-analytics architecture

> Architecture diagram of the GitHub repository salveendutt/big-data-analytics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/salveendutt/big-data-analytics
- Repository: https://github.com/salveendutt/big-data-analytics

## Overview

##### Data Sources Layer
- Multiple input data sources (Kaggle and OpenML datasets)

##### Data Ingestion Layer
- Apache NiFi components (two instances visible in services/nifi and services/nifi-2-0)
- Data flow controllers and processors
- Connection to both streaming and batch pipelines

##### Processing Layer (split into two parallel paths)
a. Batch Processing:
- Batch processing service (from services/batch_processing)
- Connection to data storage
- Batch processing workflows

b. Stream Processing:
- Streaming processing service (from services/streaming_processing)
- Streaming simulation service (from services/streaming_simulation)
- Real-time data flow paths

##### Storage Layer
- Cassandra database (visible in services/cassandra)
- Table structures defined in cassandra-table-setup.cql

- Data sources at the top
- NiFi ingestion layer below
- Parallel processing paths in the middle

##### Connection Types
- Double lines for high-bandwidth connections

##### Component Details
- Port numbers from docker-compose.yml
- Configuration dependencies

Special Considerations:

##### Dual nature of the system
- Batch processing path for historical analysis
- Stream processing path for real-time fraud detection

##### Monitoring and logging flows
- How NiFi handles data routing
- Error handling paths

##### Security Layer
- Authentication points (referenced in authorization-access-data.json)
- Authorized access paths

##### Scalability Aspects
- Containerized services
- Potential scaling points

- Symbols for different service types
- Container vs. external service indicators

Level of Detail:
- Configuration relationships without overwhelming detail

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Data Sources"
        DS1[("Kaggle Dataset")]:::datasource
        DS2[("OpenML Dataset")]:::datasource
        EDA["EDA Notebooks"]:::notebook
    end

    subgraph "Data Ingestion Layer"
        NF1["Primary NiFi Instance"]:::nifi
        NF2["Secondary NiFi Instance"]:::nifi
        NFC["NiFi Configuration"]:::config
        NFF["NiFi Flow Definition"]:::config
    end

    subgraph "Processing Layer"
        subgraph "Batch Processing"
            BPM["Batch Processing Main"]:::processing
            BPC["Batch Processing Config"]:::config
            BPU["Batch Processing Utils"]:::utils
        end

        subgraph "Stream Processing"
            SPM["Stream Processing Main"]:::processing
            SPC["Stream Processing Config"]:::config
            SSM["Streaming Simulation"]:::processing
        end
    end

    subgraph "Storage Layer"
        CDB[("Cassandra DB")]:::database
        CTS["Cassandra Table Setup"]:::config
    end

    subgraph "Security Layer"
        AD["Authorization Data"]:::security
        NSC["NiFi Security Config"]:::security
        NUM["NiFi User Management"]:::security
    end

    %% Data Flow Connections
    DS1 --> NF1
    DS2 --> NF1
    NF1 --> BPM
    NF1 --> SPM
    NF2 --> BPM
    NF2 --> SPM
    BPM --> CDB
    SPM --> CDB
    SSM --> SPM

    %% Configuration Connections
    NFC -.-> NF1
    NFF -.-> NF1
    BPC -.-> BPM
    SPC -.-> SPM
    CTS -.-> CDB
    AD -.-> NF1
    NSC -.-> NF1
    NUM -.-> NF1

    %% Click Events
    click EDA "https://github.com/salveendutt/Big-Data-Analytics/blob/main/eda/Credit Card Fraud Detection_Kaggle.ipynb"
    click SSM "https://github.com/salveendutt/Big-Data-Analytics/blob/main/services/streaming_simulation/streaming_simulation.py"
    click NF1 "https://github.com/salveendutt/Big-Data-Analytics/tree/main/services/nifi"
    click NF2 "https://github.com/salveendutt/Big-Data-Analytics/tree/main/services/nifi-2-0"
    click NFC "https://github.com/salveendutt/Big-Data-Analytics/blob/main/services/nifi/conf/nifi.properties"
    click NFF "https://github.com/salveendutt/Big-Data-Analytics/blob/main/services/nifi/conf/flow.json.gz"
    click BPM "https://github.com/salveendutt/Big-Data-Analytics/blob/main/services/batch_processing/batch_processing.py"
    click BPC "https://github.com/salveendutt/Big-Data-Analytics/blob/main/services/batch_processing/config_batch_processing.py"
    click BPU "https://github.com/salveendutt/Big-Data-Analytics/blob/main/services/batch_processing/utils_batch_processing.py"
    click SPM "https://github.com/salveendutt/Big-Data-Analytics/blob/main/services/streaming_processing/streaming_processing.py"
    click SPC "https://github.com/salveendutt/Big-Data-Analytics/blob/main/services/streaming_processing/config_streaming_processing.py"
    click CTS "https://github.com/salveendutt/Big-Data-Analytics/blob/main/services/cassandra/init-cql/cassandra-table-setup.cql"
    click AD "https://github.com/salveendutt/Big-Data-Analytics/blob/main/authorization-access-data.json"
    click NSC "https://github.com/salveendutt/Big-Data-Analytics/blob/main/services/nifi/conf/authorizers.xml"
    click NUM "https://github.com/salveendutt/Big-Data-Analytics/blob/main/services/nifi/conf/users.xml"

    %% Styling
    classDef datasource fill:#3498db,stroke:#2980b9,stroke-width:2px
    classDef nifi fill:#2ecc71,stroke:#27ae60,stroke-width:2px
    classDef processing fill:#f1c40f,stroke:#f39c12,stroke-width:2px
    classDef database fill:#9b59b6,stroke:#8e44ad,stroke-width:2px
    classDef config fill:#95a5a6,stroke:#7f8c8d,stroke-width:2px
    classDef security fill:#e74c3c,stroke:#c0392b,stroke-width:2px
    classDef utils fill:#1abc9c,stroke:#16a085,stroke-width:2px
    classDef notebook fill:#34495e,stroke:#2c3e50,stroke-width:2px
```
