How salveendutt/big-data-analytics works
Data Sources Layer
• Multiple input data sources (Kaggle and OpenML datasets)
Data Ingestion Layer
• Apache NiFi components (two instances visible in services/nifi and services/nifi-2-0)
• Data flow controllers and processors
• Connection to both streaming and batch pipelines
Processing Layer (split into two parallel paths)
a. Batch Processing:
• Batch processing service (from services/batch_processing)
• Connection to data storage
• Batch processing workflows
b. Stream Processing:
• Streaming processing service (from services/streaming_processing)
• Streaming simulation service (from services/streaming_simulation)
• Real-time data flow paths
Storage Layer
• Cassandra database (visible in services/cassandra)
• Table structures defined in cassandra-table-setup.cql
• Data sources at the top
• NiFi ingestion layer below
• Parallel processing paths in the middle
Connection Types
• Double lines for high-bandwidth connections
Component Details
• Port numbers from docker-compose.yml
• Configuration dependencies
Special Considerations:
Dual nature of the system
• Batch processing path for historical analysis
• Stream processing path for real-time fraud detection
Monitoring and logging flows
• How NiFi handles data routing
• Error handling paths
Security Layer
• Authentication points (referenced in authorization-access-data.json)
• Authorized access paths
Scalability Aspects
• Containerized services
• Potential scaling points
• Symbols for different service types
• Container vs. external service indicators
Level of Detail:
• Configuration relationships without overwhelming detail