How makemytrip/datashark works
dataShark is a Spark-based framework for batch and streaming security/network analytics. Users configure a use case and choose an input, analysis module, and output; the runtime loads the configuration, obtains data, invokes the module, and dispatches results to an output plugin. The README also documents production deployments with Fluentd acquisition, Kafka queuing, Elasticsearch/HDFS persistence, and ElastAlert alerting. Those integrations are shown as documented capabilities; source excerpts do not establish all their runtime wiring.
Main entry points: datashark.py Production Runner · datashark_standalone.py Standalone Runner · out_csv.py CSV Output · out_elasticsearch.py Elasticsearch Output
Components 21
Spark Runtime
- Standalone Runner
datashark_standalone.py - Production Runner
datashark.py - Use-case Configs
conf - Apache Spark
- Files / HDFS Input
- Log Filtering
datashark.py
Use Cases
- Web Anomaly Detection
conf/web-anomaly-detection/web-anomaly-detection.py - Word Count
conf/wordcount/wordcount.py - Custom Use Case
conf
Output Plugins
- CSV Output
plugins/output/out_csv.py - Syslog Output
plugins/output/out_syslog.py - Elasticsearch Output
plugins/output/out_elasticsearch.py
Production Services
- Elasticsearch
- HDFS
- Fluentd Acquisition
- Kafka Queue
- ElastAlert
Other components
- Analyst / Operator
- CSV Filesystem
- Syslog Server
- Alert Consumers
Connections 32
- Analyst / Operator → Standalone Runner: runs batch
- Analyst / Operator → Production Runner: runs production
- Analyst / Operator → Use-case Configs: configures
- Standalone Runner → Use-case Configs: loads config
- Production Runner → Use-case Configs: loads config
- Standalone Runner → Apache Spark: creates context
- Production Runner → Apache Spark: creates context
- Standalone Runner → Files / HDFS Input: reads batch data
- Production Runner → Files / HDFS Input: reads batch data
- Production Runner → Kafka Queue: creates stream
- Standalone Runner → Web Anomaly Detection: invokes loader
- Standalone Runner → Word Count: invokes loader
- Production Runner → Web Anomaly Detection: invokes loader
- Production Runner → Word Count: invokes loader
- Production Runner → Log Filtering: filters stream
- Production Runner → Custom Use Case: invokes configured code
- Standalone Runner → Custom Use Case: invokes configured code
- Standalone Runner → CSV Output: dispatches results
- Standalone Runner → Syslog Output: dispatches results
- Standalone Runner → Elasticsearch Output: dispatches results
- Production Runner → CSV Output: dispatches results
- Production Runner → Syslog Output: dispatches results
- Production Runner → Elasticsearch Output: dispatches results
- CSV Output → CSV Filesystem: writes rows
- Syslog Output → Syslog Server: sends events
- Elasticsearch Output → Elasticsearch: writes documents
- Fluentd Acquisition → Kafka Queue: forwards events
- HDFS → Files / HDFS Input: provides input
- Elasticsearch → ElastAlert: monitored by
- ElastAlert → Alert Consumers: generates alerts
- Elasticsearch Output → HDFS: may persist
- CSV Output → HDFS: may persist