# cyb3rward0g/helk architecture

> Architecture diagram of the GitHub repository cyb3rward0g/helk, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cyb3rward0g/helk
- Repository: https://github.com/cyb3rward0g/helk

## Overview

##### Project type and purpose
- Project type: Full-stack data-pipeline & analytics platform deployed via Docker microservices
- Purpose: An open-source threat-hunting platform (HELK) with ELK stack ingestion, SQL/Graph analytics (Spark & KSQL), alerting (Elastalert), and Jupyter-based machine-learning demos

##### File structure
- Top-level “docker/” directory holds one folder per service (helk-elasticsearch, helk-logstash, helk-kibana, helk-kafka-broker, helk-zookeeper, helk-spark-master/worker, helk-jupyter, helk-elastalert, helk-nginx, plus base images)
- “configs/” holds Beats (filebeat, winlogbeat) and Zeek shipping configs
- “docs/architecture/” contains architecture write-ups for ES, Kibana, Logstash
- Scripts at root orchestrate container lifecycle (install, update, remove)

##### README
- Core components: ELK stack (Elasticsearch, Logstash, Kibana), Apache Kafka & Zookeeper, KSQL, Apache Spark cluster, Jupyter notebooks, Elastalert
- Goals: threat hunting with structured streaming, SQL, graph analytics

##### Detail
- Data-flows (Beats → Logstash → Kafka → ES, etc.)
- How Jupyter connects to Spark cluster and ES
- Reference config files (e.g., filebeat.yml) as annotations if needed

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Ingestion Layer
    subgraph "Ingestion Layer"
        direction TB
        Filebeat["Filebeat"]:::ingestion
        Winlogbeat["Winlogbeat"]:::ingestion
        ZeekShipper["Zeek Shipper"]:::ingestion
        Logstash["Logstash Ingestion Pipeline"]:::ingestion
    end

    %% Streaming Layer
    subgraph "Streaming Layer"
        direction TB
        Zookeeper["Zookeeper Coordination Service"]:::infra
        KafkaBase["Kafka Base Image"]:::infra
        KafkaBroker["Apache Kafka Broker"]:::streaming
        KSQL["KSQL Streaming SQL Engine"]:::streaming
    end

    %% Storage Layer
    subgraph "Storage Layer"
        direction TB
        Elasticsearch["Elasticsearch Storage Cluster"]:::storage
    end

    %% Processing Layer
    subgraph "Processing Layer"
        direction TB
        SparkBase["Spark Base Image"]:::infra
        SparkMaster["Apache Spark Master"]:::processing
        SparkWorker["Apache Spark Worker"]:::processing
        Jupyter["JupyterLab Notebooks & ML Demos"]:::processing
    end

    %% UI Layer
    subgraph "UI Layer"
        direction TB
        Kibana["Kibana Visualization UI"]:::ui
        Elastalert["Elastalert Alerting Engine"]:::ui
    end

    %% Infrastructure Layer
    subgraph "Infrastructure"
        direction TB
        Nginx["Nginx Reverse Proxy"]:::infra
        Orchestration["Container Orchestration Scripts"]:::infra
    end

    %% Data Flows
    Filebeat -->|"JSON over TCP"| Logstash
    Winlogbeat -->|"JSON over TCP"| Logstash
    ZeekShipper -->|"JSON over TCP"| Logstash

    Logstash -->|"Kafka Topics"| KafkaBroker

    KafkaBroker -->|"writes to Index"| Elasticsearch
    KafkaBroker -->|"streams to"| KSQL
    KSQL -->|"outputs to Kafka/ES"| KafkaBroker
    Elastalert -->|"polls"| Elasticsearch
    Elasticsearch -->|"alerts"| Elastalert

    SparkMaster -->|"reads from ES/Kafka"| Elasticsearch
    SparkWorker -->|"reads from ES/Kafka"| KafkaBroker
    Jupyter -->|"connects"| SparkMaster
    Jupyter -->|"queries"| Elasticsearch

    Kibana -->|"queries"| Elasticsearch

    Nginx -->|"routes HTTP to Kibana"| Kibana
    Nginx -->|"routes HTTP to Jupyter"| Jupyter
    Nginx -->|"routes HTTP to Elastalert UI"| Elastalert

    %% Infrastructure connections
    Zookeeper -->|"coordinates"| KafkaBroker
    SparkBase -->|"builds"| SparkMaster
    SparkBase -->|"builds"| SparkWorker
    KafkaBase -->|"builds"| KafkaBroker

    %% Click Events
    click Filebeat "https://github.com/cyb3rward0g/helk/blob/master/configs/filebeat/filebeat-osquery.yml"
    click Winlogbeat "https://github.com/cyb3rward0g/helk/blob/master/configs/winlogbeat/winlogbeat.yml"
    click ZeekShipper "https://github.com/cyb3rward0g/helk/blob/master/configs/filebeat/filebeat-zeek.yml"
    click Logstash "https://github.com/cyb3rward0g/helk/tree/master/docker/helk-logstash"
    click KafkaBroker "https://github.com/cyb3rward0g/helk/tree/master/docker/helk-kafka-broker"
    click KafkaBase "https://github.com/cyb3rward0g/helk/tree/master/docker/helk-kafka-base/Dockerfile"
    click Zookeeper "https://github.com/cyb3rward0g/helk/tree/master/docker/helk-zookeeper"
    click KSQL "https://github.com/cyb3rward0g/helk/blob/master/docker/helk-ksql/sysmon-join.commands"
    click Elasticsearch "https://github.com/cyb3rward0g/helk/tree/master/docker/helk-elasticsearch"
    click Kibana "https://github.com/cyb3rward0g/helk/tree/master/docker/helk-kibana"
    click Elastalert "https://github.com/cyb3rward0g/helk/tree/master/docker/helk-elastalert"
    click SparkMaster "https://github.com/cyb3rward0g/helk/tree/master/docker/helk-spark-master"
    click SparkWorker "https://github.com/cyb3rward0g/helk/tree/master/docker/helk-spark-worker"
    click SparkBase "https://github.com/cyb3rward0g/helk/tree/master/docker/helk-spark-base/Dockerfile"
    click Jupyter "https://github.com/cyb3rward0g/helk/tree/master/docker/helk-jupyter"
    click Nginx "https://github.com/cyb3rward0g/helk/tree/master/docker/helk-nginx"
    click Orchestration "https://github.com/cyb3rward0g/helk/blob/master/docker/helk_install.sh"

    %% Styles
    classDef ingestion fill:#bbdefb,stroke:#1e88e5,color:#1e88e5;
    classDef streaming fill:#ffcc80,stroke:#fb8c00,color:#e65100;
    classDef storage fill:#c8e6c9,stroke:#43a047,color:#1b5e20;
    classDef processing fill:#e1bee7,stroke:#8e24aa,color:#4a148c;
    classDef ui fill:#fff9c4,stroke:#fdd835,color:#f57f17;
    classDef infra fill:#eeeeee,stroke:#757575,color:#424242;
```
