# juhong-namgung/real-time-malicious-url-detection-system-using-deep-learning architecture

> Architecture diagram of the GitHub repository juhong-namgung/real-time-malicious-url-detection-system-using-deep-learning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/juhong-namgung/real-time-malicious-url-detection-system-using-deep-learning
- Repository: https://github.com/juhong-namgung/real-time-malicious-url-detection-system-using-deep-learning

## Overview

##### Data Ingestion Layer
- Kafka Producer (FileProducer.java)
- Twitter Spout (TwitterSpout.java)

##### Stream Processing Layer (Apache Storm Topology)
- ExtractionURLBolt
- ValidationURLBolt
- ExpansionURLBolt
- DetectionBolt

##### Machine Learning Component
- TensorFlow/Keras Model (saved_model.pb)
- This as a separate service that interfaces with the DetectionBolt

##### Storage Components
- Kafka message queue
- Model storage (models directory)

##### Middle Level (Processing)
- Storm topology components in the middle
- Bolts in sequential order from left to right
- Queue symbols between components

##### Bottom Level (ML & Storage)
- ML model component
- Model storage
- Output handlers

Specific Elements to Include:

##### Component Relationships
- Arrowheads to show data flow direction

##### Processing Units
- Use rectangles for bolts
- Use circles for spouts

##### Storage Components: Orange

Additional Considerations:

##### Scalability Indicators
- Parallel processing notation for Storm bolts
- Potential scaling points

##### Error Handling
- Error flow paths
- Retry mechanisms

##### Monitoring Points
- Monitoring indicators
- Logging points

- Any abbreviations used

Level of Detail:
- Timing and latency expectations
- System boundaries

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Sources
    subgraph Sources
        KP["Kafka Producer"]:::source
        TS["Twitter Spout"]:::source
    end

    %% Stream Processing Layer
    subgraph "Storm Topology"
        MT["Main Topology"]:::storm
        EX["Extraction URL Bolt"]:::bolt
        VB["Validation URL Bolt"]:::bolt
        EB["Expansion URL Bolt"]:::bolt
        DB["Detection Bolt"]:::bolt
    end

    %% Machine Learning Layer
    subgraph "ML Components"
        TF["TensorFlow Model"]:::ml
        MS[("Model Storage")]:::storage
    end

    %% Storage Layer
    subgraph "Storage Components"
        KQ[("Kafka Queue")]:::queue
        DS[("Data Storage")]:::storage
    end

    %% Data Processing Utilities
    subgraph "Processing Utilities"
        DP["Data Preprocessor"]:::util
        ME["Model Evaluator"]:::util
        MS2["Model Saver"]:::util
    end

    %% Relationships
    KP --> |data| KQ
    TS --> |tweets| MT
    KQ --> |messages| MT
    MT --> |urls| EX
    EX --> |extracted| VB
    VB --> |validated| EB
    EB --> |expanded| DB
    DB <--> |inference| TF
    TF --> |results| DS
    MS --> |load| TF
    DP --> |processed data| TF
    ME --> |evaluation| TF
    MS2 --> |save| MS

    %% Click Events
    click KP "https://github.com/Juhong-Namgung/Real-time-Malicious-URL-Detection-System-using-Deep-Learning/blob/master/Real-time-Detection-using-Storm/src/main/java/main/kafka/FileProducer.java"
    click TS "https://github.com/Juhong-Namgung/Real-time-Malicious-URL-Detection-System-using-Deep-Learning/blob/master/Real-time-Detection-using-Storm/src/main/java/main/tta/perf/TwitterSpout.java"
    click EX "https://github.com/Juhong-Namgung/Real-time-Malicious-URL-Detection-System-using-Deep-Learning/blob/master/Real-time-Detection-using-Storm/src/main/java/main/dke/detectURLtopo/ExtractionURLBolt.java"
    click VB "https://github.com/Juhong-Namgung/Real-time-Malicious-URL-Detection-System-using-Deep-Learning/blob/master/Real-time-Detection-using-Storm/src/main/java/main/dke/detectURLtopo/ValidationURLBolt.java"
    click EB "https://github.com/Juhong-Namgung/Real-time-Malicious-URL-Detection-System-using-Deep-Learning/blob/master/Real-time-Detection-using-Storm/src/main/java/main/dke/detectURLtopo/ExpansionURLBolt.java"
    click DB "https://github.com/Juhong-Namgung/Real-time-Malicious-URL-Detection-System-using-Deep-Learning/blob/master/Real-time-Detection-using-Storm/src/main/java/main/dke/detectURLtopo/DetectionBolt.java"
    click MT "https://github.com/Juhong-Namgung/Real-time-Malicious-URL-Detection-System-using-Deep-Learning/blob/master/Real-time-Detection-using-Storm/src/main/java/main/dke/detectURLtopo/MainTopo.java"
    click TF "https://github.com/Juhong-Namgung/Real-time-Malicious-URL-Detection-System-using-Deep-Learning/blob/master/Real-time-Detection-using-Storm/src/main/resources/models/cnn/saved_model.pb"
    click MS "https://github.com/Juhong-Namgung/Real-time-Malicious-URL-Detection-System-using-Deep-Learning/tree/master/Malicious-URL-Detection-using-Deep-Learning/models/"
    click DS "https://github.com/Juhong-Namgung/Real-time-Malicious-URL-Detection-System-using-Deep-Learning/tree/master/Malicious-URL-Detection-using-Deep-Learning/data/"
    click DP "https://github.com/Juhong-Namgung/Real-time-Malicious-URL-Detection-System-using-Deep-Learning/blob/master/Malicious-URL-Detection-using-Deep-Learning/models/data_preprocessor.py"
    click ME "https://github.com/Juhong-Namgung/Real-time-Malicious-URL-Detection-System-using-Deep-Learning/blob/master/Malicious-URL-Detection-using-Deep-Learning/models/model_evaluator.py"
    click MS2 "https://github.com/Juhong-Namgung/Real-time-Malicious-URL-Detection-System-using-Deep-Learning/blob/master/Malicious-URL-Detection-using-Deep-Learning/models/model_saver.py"

    %% Styles
    classDef source fill:#2374f7,stroke:#000,stroke-width:2px
    classDef storm fill:#48a842,stroke:#000,stroke-width:2px
    classDef bolt fill:#48a842,stroke:#000,stroke-width:1px
    classDef ml fill:#9d3fd4,stroke:#000,stroke-width:2px
    classDef storage fill:#f7a223,stroke:#000,stroke-width:2px
    classDef queue fill:#f7a223,stroke:#000,stroke-width:1px
    classDef util fill:#23b7f7,stroke:#000,stroke-width:1px

    %% Legend
    subgraph Legend
        L1["Data Source"]:::source
        L2["Storm Component"]:::storm
        L3["ML Component"]:::ml
        L4["Storage"]:::storage
        L5["Utility"]:::util
    end
```
