# g-lehmann/concerto-siem architecture

> Architecture diagram of the GitHub repository g-lehmann/concerto-siem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/g-lehmann/concerto-siem
- Repository: https://github.com/g-lehmann/concerto-siem

## Overview

#### Purpose and deployment entry points
Concerto is a containerized SIEM for IDMEFv2 cyber/physical alerts and syslog-derived detections. Operators use `make up`/`make down`; composition and service wiring are defined in `docker-compose.yml`, with a test stack in `docker-compose-test.yml`. Kubernetes/Podman deployment manifests are also provided under `pods/`, notably `pods/logstash.yml` and `pods/gui.yml`.

#### Ingestion boundary
Logstash is the external ingestion gateway: it accepts syslog over TCP/UDP 6514 and IDMEFv2 JSON over HTTP 4690. Pipeline definitions separate collected, analyzed, cleaned, IDMEFv2, processed, stored, and error streams in `logstash/pipelines.yml`, `logstash/pipeline/00-my_collected.conf`, and `logstash/pipeline/00-my_collected_idmefv2.conf`. This is the main boundary between untrusted input and normalized internal events.

#### Log normalization and detection
Incoming logs are parsed into IDMEFv2-oriented fields, enriched, and matched against YAML rulesets. The Ruby transformation/matching layer resides in `logstash/scripts/` (for example `logstash/scripts/rules.rb`, `logstash/scripts/extraction.rb`, `logstash/scripts/fill_alert.rb`). Detection content is split deliberately: source-specific Grok parsing rules in `logstash/rulesets/ssh.yml` and IDMEFv2 alert construction rules in `logstash/idmef/ssh.yml`; matching IDs join the two stages.

#### Kafka event bus
Kafka is the asynchronous communication backbone between ingestion, storage, and correlation services, with ZooKeeper coordination. Container definitions are `kafka/Dockerfile` and `zoo/Dockerfile`; correlator Kafka connection settings are in `correlator/kafka.yaml`. Kafdrop (`kafdrop/Dockerfile`) exposes broker/topic observability. Kafka decouples producers from downstream consumers and is the key operational dependency for event delivery.

#### Storage and schema initialization
Elasticsearch stores JSON alerts and logs and becomes queryable once data creates its indexes. Its runtime configuration is in `elasticsearch/Dockerfile` and `elasticsearch/config/elasticsearch.yml`. A one-shot setup container applies IDMEFv2 mappings via `setup/entrypoint.sh` and `setup/config/IDMEFv2.mapping`. PostgreSQL (`postgres/Dockerfile`) supports the Prewikka-based GUI’s relational state.

#### Correlation engine
The correlator consumes event data through Kafka and produces higher-level detections using ElastAlert-derived/YARA-style rules. Its runtime launcher and configuration are `correlator/concerto_correlator.sh`, `correlator/config.yaml`, and `correlator/elastalert_concerto/concerto.py`; example correlation content is `correlator/rules/bruteforce.yml`. This is distinct from Logstash’s single-log parsing: it performs cross-event correlation.

#### Presentation and test traffic
The GUI is a Prewikka OSS deployment configured for IDMEFv2 views and backed by Elasticsearch/PostgreSQL: `gui/entrypoint.sh`, `gui/config/prewikka.conf`, and `gui/config/idmefv2.conf`. NGINX is an optional log-generating test source (`nginx/nginx.conf`, `nginx/Dockerfile`) that exercises the same syslog ingestion path. Tests inject representative logs and IDMEFv2 documents through `tests/entrypoint_tests_logs.sh` and `tests/entrypoint_tests_idmefv2.py`.

Main entry points: [`kafka/Dockerfile`](https://github.com/g-lehmann/concerto-siem/blob/main/kafka/Dockerfile) (Kafka), [`logstash/pipelines.yml`](https://github.com/g-lehmann/concerto-siem/blob/main/logstash/pipelines.yml) (Logstash gateway), [`docker-compose.yml`](https://github.com/g-lehmann/concerto-siem/blob/main/docker-compose.yml) (Container stack), [`logstash/scripts/rules.rb`](https://github.com/g-lehmann/concerto-siem/blob/main/logstash/scripts/rules.rb) (Transform & matching)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_deploy["Deployment"]
  node_compose["Container stack<br/>Docker Compose<br/>[docker-compose.yml]"]
end

subgraph group_ingest["Ingestion &amp; Detection"]
  node_logstash["Logstash gateway<br/>ingestion service<br/>[pipelines.yml]"]
  node_syslog_pipeline["Syslog pipeline"]
  node_idmef_pipeline["IDMEFv2 pipeline"]
  node_transform_rules["Transform &amp; matching<br/>Ruby rules layer<br/>[rules.rb]"]
  node_grok_rules["Source parsing rules<br/>YAML ruleset<br/>[ssh.yml]"]
  node_idmef_rules["Alert construction rules<br/>YAML ruleset<br/>[ssh.yml]"]
  node_nginx["NGINX test source<br/>log generator"]
  node_test_logs["Log test injector<br/>test traffic"]
  node_test_idmef["IDMEF test injector<br/>test traffic"]
end

subgraph group_eventing["Event Backbone"]
  node_kafka["Kafka<br/>event bus"]
  node_zookeeper["ZooKeeper<br/>Kafka coordination"]
  node_kafdrop["Kafdrop<br/>broker observability"]
end

subgraph group_storage["Storage"]
  node_elasticsearch[("Elasticsearch<br/>alert and log store")]
  node_mapping_setup["IDMEF mapping setup<br/>one-shot initializer<br/>[entrypoint.sh]"]
  node_postgres[("PostgreSQL<br/>GUI state store")]
end

subgraph group_analysis["Correlation &amp; UI"]
  node_correlator["Correlation engine<br/>cross-event detection"]
  node_correlation_rules["Correlation rules<br/>ElastAlert/YARA rules<br/>[bruteforce.yml]"]
  node_prewikka["Prewikka GUI<br/>presentation service<br/>[entrypoint.sh]"]
end

node_compose -->|"runs"| node_logstash
node_compose -->|"runs"| node_kafka
node_compose -->|"runs"| node_elasticsearch
node_compose -->|"runs"| node_correlator
node_compose -->|"runs"| node_prewikka
node_nginx -->|"syslog"| node_logstash
node_test_logs -.->|"syslog"| node_logstash
node_test_idmef -.->|"IDMEFv2 HTTP"| node_logstash
node_logstash -->|"syslog input"| node_syslog_pipeline
node_logstash -->|"JSON input"| node_idmef_pipeline
node_syslog_pipeline -->|"parsed events"| node_transform_rules
node_idmef_pipeline -->|"normalized events"| node_transform_rules
node_grok_rules -.->|"parsing patterns"| node_transform_rules
node_idmef_rules -.->|"alert definitions"| node_transform_rules
node_transform_rules -->|"normalized events"| node_kafka
node_zookeeper -->|"coordinates"| node_kafka
node_kafka -.->|"observed by"| node_kafdrop
node_kafka -->|"stored events"| node_elasticsearch
node_mapping_setup -->|"installs mappings"| node_elasticsearch
node_kafka -->|"consumed events"| node_correlator
node_correlation_rules -.->|"detection logic"| node_correlator
node_correlator -->|"correlated detections"| node_kafka
node_elasticsearch -->|"alert and log queries"| node_prewikka
node_postgres -->|"relational state"| node_prewikka

click node_compose "https://github.com/g-lehmann/concerto-siem/blob/main/docker-compose.yml"
click node_logstash "https://github.com/g-lehmann/concerto-siem/blob/main/logstash/pipelines.yml"
click node_syslog_pipeline "https://github.com/g-lehmann/concerto-siem/blob/main/logstash/pipeline/00-my_collected.conf"
click node_idmef_pipeline "https://github.com/g-lehmann/concerto-siem/blob/main/logstash/pipeline/00-my_collected_idmefv2.conf"
click node_transform_rules "https://github.com/g-lehmann/concerto-siem/blob/main/logstash/scripts/rules.rb"
click node_grok_rules "https://github.com/g-lehmann/concerto-siem/blob/main/logstash/rulesets/ssh.yml"
click node_idmef_rules "https://github.com/g-lehmann/concerto-siem/blob/main/logstash/idmef/ssh.yml"
click node_kafka "https://github.com/g-lehmann/concerto-siem/blob/main/kafka/Dockerfile"
click node_zookeeper "https://github.com/g-lehmann/concerto-siem/blob/main/zoo/Dockerfile"
click node_kafdrop "https://github.com/g-lehmann/concerto-siem/blob/main/kafdrop/Dockerfile"
click node_elasticsearch "https://github.com/g-lehmann/concerto-siem/blob/main/elasticsearch/Dockerfile"
click node_mapping_setup "https://github.com/g-lehmann/concerto-siem/blob/main/setup/entrypoint.sh"
click node_postgres "https://github.com/g-lehmann/concerto-siem/blob/main/postgres/Dockerfile"
click node_correlator "https://github.com/g-lehmann/concerto-siem/blob/main/correlator/concerto_correlator.sh"
click node_correlation_rules "https://github.com/g-lehmann/concerto-siem/blob/main/correlator/rules/bruteforce.yml"
click node_prewikka "https://github.com/g-lehmann/concerto-siem/blob/main/gui/entrypoint.sh"
click node_nginx "https://github.com/g-lehmann/concerto-siem/blob/main/nginx/Dockerfile"
click node_test_logs "https://github.com/g-lehmann/concerto-siem/blob/main/tests/entrypoint_tests_logs.sh"
click node_test_idmef "https://github.com/g-lehmann/concerto-siem/blob/main/tests/entrypoint_tests_idmefv2.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_compose toneBlue
class node_logstash,node_syslog_pipeline,node_idmef_pipeline,node_transform_rules,node_grok_rules,node_idmef_rules,node_nginx,node_test_logs,node_test_idmef toneAmber
class node_kafka,node_zookeeper,node_kafdrop toneMint
class node_elasticsearch,node_mapping_setup,node_postgres toneRose
class node_correlator,node_correlation_rules,node_prewikka toneIndigo
```

## Components

### Deployment

- [Container stack](https://github.com/g-lehmann/concerto-siem/blob/main/docker-compose.yml): Docker Compose · Primary operator entry point for the SIEM stack. · `docker-compose.yml`

### Ingestion & Detection

- [Logstash gateway](https://github.com/g-lehmann/concerto-siem/blob/main/logstash/pipelines.yml): ingestion service · Accepts syslog and IDMEFv2 HTTP input. · `logstash/pipelines.yml`
- [Syslog pipeline](https://github.com/g-lehmann/concerto-siem/blob/main/logstash/pipeline/00-my_collected.conf): pipeline · Separates collected, analyzed, cleaned, processed, stored, and error streams. · `logstash/pipeline/00-my_collected.conf`
- [IDMEFv2 pipeline](https://github.com/g-lehmann/concerto-siem/blob/main/logstash/pipeline/00-my_collected_idmefv2.conf): pipeline · Processes incoming IDMEFv2 JSON documents. · `logstash/pipeline/00-my_collected_idmefv2.conf`
- [Transform & matching](https://github.com/g-lehmann/concerto-siem/blob/main/logstash/scripts/rules.rb): Ruby rules layer · Normalizes fields, enriches events, and constructs alerts. · `logstash/scripts/rules.rb`
- [Source parsing rules](https://github.com/g-lehmann/concerto-siem/blob/main/logstash/rulesets/ssh.yml): YAML ruleset · Source-specific Grok extraction rules. · `logstash/rulesets/ssh.yml`
- [Alert construction rules](https://github.com/g-lehmann/concerto-siem/blob/main/logstash/idmef/ssh.yml): YAML ruleset · IDMEFv2 alert rules joined to parsing matches by ID. · `logstash/idmef/ssh.yml`
- [NGINX test source](https://github.com/g-lehmann/concerto-siem/blob/main/nginx/Dockerfile): log generator · Optional source exercising syslog ingestion. · `nginx/Dockerfile`
- [Log test injector](https://github.com/g-lehmann/concerto-siem/blob/main/tests/entrypoint_tests_logs.sh): test traffic · Injects representative syslog records. · `tests/entrypoint_tests_logs.sh`
- [IDMEF test injector](https://github.com/g-lehmann/concerto-siem/blob/main/tests/entrypoint_tests_idmefv2.py): test traffic · Injects representative IDMEFv2 documents. · `tests/entrypoint_tests_idmefv2.py`

### Event Backbone

- [Kafka](https://github.com/g-lehmann/concerto-siem/blob/main/kafka/Dockerfile): event bus · Asynchronous backbone for ingestion, storage, and correlation. · `kafka/Dockerfile`
- [ZooKeeper](https://github.com/g-lehmann/concerto-siem/blob/main/zoo/Dockerfile): Kafka coordination · `zoo/Dockerfile`
- [Kafdrop](https://github.com/g-lehmann/concerto-siem/blob/main/kafdrop/Dockerfile): broker observability · Exposes Kafka topics and broker state. · `kafdrop/Dockerfile`

### Storage

- [Elasticsearch](https://github.com/g-lehmann/concerto-siem/blob/main/elasticsearch/Dockerfile): alert and log store · Stores queryable JSON alerts and logs. · `elasticsearch/Dockerfile`
- [IDMEF mapping setup](https://github.com/g-lehmann/concerto-siem/blob/main/setup/entrypoint.sh): one-shot initializer · Applies IDMEFv2 mappings before indexed use. · `setup/entrypoint.sh`
- [PostgreSQL](https://github.com/g-lehmann/concerto-siem/blob/main/postgres/Dockerfile): GUI state store · Provides relational state for Prewikka. · `postgres/Dockerfile`

### Correlation & UI

- [Correlation engine](https://github.com/g-lehmann/concerto-siem/blob/main/correlator/concerto_correlator.sh): cross-event detection · Consumes event streams and emits higher-level detections. · `correlator/concerto_correlator.sh`
- [Correlation rules](https://github.com/g-lehmann/concerto-siem/blob/main/correlator/rules/bruteforce.yml): ElastAlert/YARA rules · Defines multi-event detections such as brute force. · `correlator/rules/bruteforce.yml`
- [Prewikka GUI](https://github.com/g-lehmann/concerto-siem/blob/main/gui/entrypoint.sh): presentation service · Presents IDMEFv2 views over Elasticsearch and PostgreSQL. · `gui/entrypoint.sh`

## Connections

- Container stack → Logstash gateway: runs
- Container stack → Kafka: runs
- Container stack → Elasticsearch: runs
- Container stack → Correlation engine: runs
- Container stack → Prewikka GUI: runs
- NGINX test source → Logstash gateway: syslog
- Log test injector → Logstash gateway: syslog
- IDMEF test injector → Logstash gateway: IDMEFv2 HTTP
- Logstash gateway → Syslog pipeline: syslog input
- Logstash gateway → IDMEFv2 pipeline: JSON input
- Syslog pipeline → Transform & matching: parsed events
- IDMEFv2 pipeline → Transform & matching: normalized events
- Source parsing rules → Transform & matching: parsing patterns
- Alert construction rules → Transform & matching: alert definitions
- Transform & matching → Kafka: normalized events
- ZooKeeper → Kafka: coordinates
- Kafka → Kafdrop: observed by
- Kafka → Elasticsearch: stored events
- IDMEF mapping setup → Elasticsearch: installs mappings
- Kafka → Correlation engine: consumed events
- Correlation rules → Correlation engine: detection logic
- Correlation engine → Kafka: correlated detections
- Elasticsearch → Prewikka GUI: alert and log queries
- PostgreSQL → Prewikka GUI: relational state