g-lehmann/concerto-siem

Purpose and deployment entry points

Concerto is a containerized SIEM for IDMEFv2 cyber/physical alerts and syslog-derived detections. Operators use make up/make down; composition and service wiring are defined in docker-compose.yml, with a test stack in docker-compose-test.yml. Kubernetes/Podman deployment manifests are also provided under pods/, notably pods/logstash.yml and pods/gui.yml.

Ingestion boundary

Logstash is the external ingestion gateway: it accepts syslog over TCP/UDP 6514 and IDMEFv2 JSON over HTTP 4690. Pipeline definitions separate collected, analyzed, cleaned, IDMEFv2, processed, stored, and error streams in logstash/pipelines.yml, logstash/pipeline/00-my_collected.conf, and logstash/pipeline/00-my_collected_idmefv2.conf. This is the main boundary between untrusted input and normalized internal events.

Log normalization and detection

Incoming logs are parsed into IDMEFv2-oriented fields, enriched, and matched against YAML rulesets. The Ruby transformation/matching layer resides in logstash/scripts/ (for example logstash/scripts/rules.rb, logstash/scripts/extraction.rb, logstash/scripts/fill_alert.rb). Detection content is split deliberately: source-specific Grok parsing rules in logstash/rulesets/ssh.yml and IDMEFv2 alert construction rules in logstash/idmef/ssh.yml; matching IDs join the two stages.

Kafka event bus

Kafka is the asynchronous communication backbone between ingestion, storage, and correlation services, with ZooKeeper coordination. Container definitions are kafka/Dockerfile and zoo/Dockerfile; correlator Kafka connection settings are in correlator/kafka.yaml. Kafdrop (kafdrop/Dockerfile) exposes broker/topic observability. Kafka decouples producers from downstream consumers and is the key operational dependency for event delivery.

Storage and schema initialization

Elasticsearch stores JSON alerts and logs and becomes queryable once data creates its indexes. Its runtime configuration is in elasticsearch/Dockerfile and elasticsearch/config/elasticsearch.yml. A one-shot setup container applies IDMEFv2 mappings via setup/entrypoint.sh and setup/config/IDMEFv2.mapping. PostgreSQL (postgres/Dockerfile) supports the Prewikka-based GUI’s relational state.

Correlation engine

The correlator consumes event data through Kafka and produces higher-level detections using ElastAlert-derived/YARA-style rules. Its runtime launcher and configuration are correlator/concerto_correlator.sh, correlator/config.yaml, and correlator/elastalert_concerto/concerto.py; example correlation content is correlator/rules/bruteforce.yml. This is distinct from Logstash’s single-log parsing: it performs cross-event correlation.

Presentation and test traffic

The GUI is a Prewikka OSS deployment configured for IDMEFv2 views and backed by Elasticsearch/PostgreSQL: gui/entrypoint.sh, gui/config/prewikka.conf, and gui/config/idmefv2.conf. NGINX is an optional log-generating test source (nginx/nginx.conf, nginx/Dockerfile) that exercises the same syslog ingestion path. Tests inject representative logs and IDMEFv2 documents through tests/entrypoint_tests_logs.sh and tests/entrypoint_tests_idmefv2.py.