How cyb3rward0g/helk works
Project type and purpose
• Project type: Full-stack data-pipeline & analytics platform deployed via Docker microservices
• Purpose: An open-source threat-hunting platform (HELK) with ELK stack ingestion, SQL/Graph analytics (Spark & KSQL), alerting (Elastalert), and Jupyter-based machine-learning demos
File structure
• Top-level “docker/” directory holds one folder per service (helk-elasticsearch, helk-logstash, helk-kibana, helk-kafka-broker, helk-zookeeper, helk-spark-master/worker, helk-jupyter, helk-elastalert, helk-nginx, plus base images)
• “configs/” holds Beats (filebeat, winlogbeat) and Zeek shipping configs
• “docs/architecture/” contains architecture write-ups for ES, Kibana, Logstash
• Scripts at root orchestrate container lifecycle (install, update, remove)
README
• Core components: ELK stack (Elasticsearch, Logstash, Kibana), Apache Kafka & Zookeeper, KSQL, Apache Spark cluster, Jupyter notebooks, Elastalert
• Goals: threat hunting with structured streaming, SQL, graph analytics
Detail
• Data-flows (Beats → Logstash → Kafka → ES, etc.)
• How Jupyter connects to Spark cluster and ES
• Reference config files (e.g., filebeat.yml) as annotations if needed