How juhong-namgung/real-time-malicious-url-detection-system-using-deep-learning works
Data Ingestion Layer
• Kafka Producer (FileProducer.java)
• Twitter Spout (TwitterSpout.java)
Stream Processing Layer (Apache Storm Topology)
• ExtractionURLBolt
• ValidationURLBolt
• ExpansionURLBolt
• DetectionBolt
Machine Learning Component
• TensorFlow/Keras Model (saved_model.pb)
• This as a separate service that interfaces with the DetectionBolt
Storage Components
• Kafka message queue
• Model storage (models directory)
Middle Level (Processing)
• Storm topology components in the middle
• Bolts in sequential order from left to right
• Queue symbols between components
Bottom Level (ML & Storage)
• ML model component
• Model storage
• Output handlers
Specific Elements to Include:
Component Relationships
• Arrowheads to show data flow direction
Processing Units
• Use rectangles for bolts
• Use circles for spouts
Storage Components: Orange
Additional Considerations:
Scalability Indicators
• Parallel processing notation for Storm bolts
• Potential scaling points
Error Handling
• Error flow paths
• Retry mechanisms
Monitoring Points
• Monitoring indicators
• Logging points
• Any abbreviations used
Level of Detail:
• Timing and latency expectations
• System boundaries