# makemytrip/datashark architecture

> Architecture diagram of the GitHub repository makemytrip/datashark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/makemytrip/datashark
- Repository: https://github.com/makemytrip/datashark

## Overview

dataShark is a Spark-based framework for batch and streaming security/network analytics. Users configure a use case and choose an input, analysis module, and output; the runtime loads the configuration, obtains data, invokes the module, and dispatches results to an output plugin. The README also documents production deployments with Fluentd acquisition, Kafka queuing, Elasticsearch/HDFS persistence, and ElastAlert alerting. Those integrations are shown as documented capabilities; source excerpts do not establish all their runtime wiring.

Main entry points: [`datashark.py`](https://github.com/makemytrip/datashark/blob/master/datashark.py) (Production Runner), [`datashark_standalone.py`](https://github.com/makemytrip/datashark/blob/master/datashark_standalone.py) (Standalone Runner), [`plugins/output/out_csv.py`](https://github.com/makemytrip/datashark/blob/master/plugins/output/out_csv.py) (CSV Output), [`plugins/output/out_elasticsearch.py`](https://github.com/makemytrip/datashark/blob/master/plugins/output/out_elasticsearch.py) (Elasticsearch Output)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Spark Runtime"]
  node_batch_entry["Standalone Runner"]
  node_production_entry["Production Runner<br/>[datashark.py]"]
  node_usecase_config["Use-case Configs"]
  node_spark["Apache Spark"]
  node_file_input[("Files / HDFS Input")]
  node_filter["Log Filtering<br/>[datashark.py]"]
end

subgraph group_analysis["Use Cases"]
  node_web_anomaly["Web Anomaly Detection"]
  node_wordcount["Word Count<br/>[wordcount.py]"]
  node_custom_usecase["Custom Use Case"]
end

subgraph group_outputs["Output Plugins"]
  node_csv["CSV Output<br/>[out_csv.py]"]
  node_syslog_output["Syslog Output<br/>[out_syslog.py]"]
  node_elasticsearch_output["Elasticsearch Output"]
end

subgraph group_production["Production Services"]
  node_elasticsearch[("Elasticsearch")]
  node_hdfs[("HDFS")]
  node_fluentd["Fluentd Acquisition"]
  node_kafka["Kafka Queue"]
  node_elastalert["ElastAlert"]
end

node_user(("Analyst / Operator"))
node_csv_store[("CSV Filesystem")]
node_syslog_service["Syslog Server"]
node_alerts(("Alert Consumers"))

node_user -->|"runs batch"| node_batch_entry
node_user -->|"runs production"| node_production_entry
node_user -->|"configures"| node_usecase_config
node_batch_entry -->|"loads config"| node_usecase_config
node_production_entry -->|"loads config"| node_usecase_config
node_batch_entry -->|"creates context"| node_spark
node_production_entry -->|"creates context"| node_spark
node_batch_entry -->|"reads batch data"| node_file_input
node_production_entry -->|"reads batch data"| node_file_input
node_production_entry -->|"creates stream"| node_kafka
node_batch_entry -->|"invokes loader"| node_web_anomaly
node_batch_entry -->|"invokes loader"| node_wordcount
node_production_entry -->|"invokes loader"| node_web_anomaly
node_production_entry -->|"invokes loader"| node_wordcount
node_production_entry -->|"filters stream"| node_filter
node_production_entry -->|"invokes configured code"| node_custom_usecase
node_batch_entry -->|"invokes configured code"| node_custom_usecase
node_batch_entry -->|"dispatches results"| node_csv
node_batch_entry -->|"dispatches results"| node_syslog_output
node_batch_entry -->|"dispatches results"| node_elasticsearch_output
node_production_entry -->|"dispatches results"| node_csv
node_production_entry -->|"dispatches results"| node_syslog_output
node_production_entry -->|"dispatches results"| node_elasticsearch_output
node_csv -->|"writes rows"| node_csv_store
node_syslog_output -->|"sends events"| node_syslog_service
node_elasticsearch_output -->|"writes documents"| node_elasticsearch
node_fluentd -.->|"forwards events"| node_kafka
node_hdfs -.->|"provides input"| node_file_input
node_elasticsearch -.->|"monitored by"| node_elastalert
node_elastalert -.->|"generates alerts"| node_alerts
node_elasticsearch_output -.->|"may persist"| node_hdfs
node_csv -.->|"may persist"| node_hdfs

click node_batch_entry "https://github.com/makemytrip/datashark/blob/master/datashark_standalone.py"
click node_production_entry "https://github.com/makemytrip/datashark/blob/master/datashark.py"
click node_usecase_config "https://github.com/makemytrip/datashark/tree/master/conf"
click node_filter "https://github.com/makemytrip/datashark/blob/master/datashark.py"
click node_web_anomaly "https://github.com/makemytrip/datashark/blob/master/conf/web-anomaly-detection/web-anomaly-detection.py"
click node_wordcount "https://github.com/makemytrip/datashark/blob/master/conf/wordcount/wordcount.py"
click node_custom_usecase "https://github.com/makemytrip/datashark/tree/master/conf"
click node_csv "https://github.com/makemytrip/datashark/blob/master/plugins/output/out_csv.py"
click node_syslog_output "https://github.com/makemytrip/datashark/blob/master/plugins/output/out_syslog.py"
click node_elasticsearch_output "https://github.com/makemytrip/datashark/blob/master/plugins/output/out_elasticsearch.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_batch_entry,node_production_entry,node_usecase_config,node_spark,node_file_input,node_filter toneBlue
class node_web_anomaly,node_wordcount,node_custom_usecase,node_csv_store toneAmber
class node_csv,node_syslog_output,node_elasticsearch_output,node_syslog_service toneMint
class node_elasticsearch,node_hdfs,node_fluentd,node_kafka,node_elastalert toneRose
class node_user,node_alerts toneIndigo
```

## Components

### Spark Runtime

- [Standalone Runner](https://github.com/makemytrip/datashark/blob/master/datashark_standalone.py): `datashark_standalone.py`
- [Production Runner](https://github.com/makemytrip/datashark/blob/master/datashark.py): `datashark.py`
- [Use-case Configs](https://github.com/makemytrip/datashark/tree/master/conf): `conf`
- **Apache Spark**
- **Files / HDFS Input**
- [Log Filtering](https://github.com/makemytrip/datashark/blob/master/datashark.py): `datashark.py`

### Use Cases

- [Web Anomaly Detection](https://github.com/makemytrip/datashark/blob/master/conf/web-anomaly-detection/web-anomaly-detection.py): `conf/web-anomaly-detection/web-anomaly-detection.py`
- [Word Count](https://github.com/makemytrip/datashark/blob/master/conf/wordcount/wordcount.py): `conf/wordcount/wordcount.py`
- [Custom Use Case](https://github.com/makemytrip/datashark/tree/master/conf): `conf`

### Output Plugins

- [CSV Output](https://github.com/makemytrip/datashark/blob/master/plugins/output/out_csv.py): `plugins/output/out_csv.py`
- [Syslog Output](https://github.com/makemytrip/datashark/blob/master/plugins/output/out_syslog.py): `plugins/output/out_syslog.py`
- [Elasticsearch Output](https://github.com/makemytrip/datashark/blob/master/plugins/output/out_elasticsearch.py): `plugins/output/out_elasticsearch.py`

### Production Services

- **Elasticsearch**
- **HDFS**
- **Fluentd Acquisition**
- **Kafka Queue**
- **ElastAlert**

### Other components

- **Analyst / Operator**
- **CSV Filesystem**
- **Syslog Server**
- **Alert Consumers**

## Connections

- Analyst / Operator → Standalone Runner: runs batch
- Analyst / Operator → Production Runner: runs production
- Analyst / Operator → Use-case Configs: configures
- Standalone Runner → Use-case Configs: loads config
- Production Runner → Use-case Configs: loads config
- Standalone Runner → Apache Spark: creates context
- Production Runner → Apache Spark: creates context
- Standalone Runner → Files / HDFS Input: reads batch data
- Production Runner → Files / HDFS Input: reads batch data
- Production Runner → Kafka Queue: creates stream
- Standalone Runner → Web Anomaly Detection: invokes loader
- Standalone Runner → Word Count: invokes loader
- Production Runner → Web Anomaly Detection: invokes loader
- Production Runner → Word Count: invokes loader
- Production Runner → Log Filtering: filters stream
- Production Runner → Custom Use Case: invokes configured code
- Standalone Runner → Custom Use Case: invokes configured code
- Standalone Runner → CSV Output: dispatches results
- Standalone Runner → Syslog Output: dispatches results
- Standalone Runner → Elasticsearch Output: dispatches results
- Production Runner → CSV Output: dispatches results
- Production Runner → Syslog Output: dispatches results
- Production Runner → Elasticsearch Output: dispatches results
- CSV Output → CSV Filesystem: writes rows
- Syslog Output → Syslog Server: sends events
- Elasticsearch Output → Elasticsearch: writes documents
- Fluentd Acquisition → Kafka Queue: forwards events
- HDFS → Files / HDFS Input: provides input
- Elasticsearch → ElastAlert: monitored by
- ElastAlert → Alert Consumers: generates alerts
- Elasticsearch Output → HDFS: may persist
- CSV Output → HDFS: may persist