# rdegon/rdegon-siem-solution architecture

> Architecture diagram of the GitHub repository rdegon/rdegon-siem-solution, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rdegon/rdegon-siem-solution
- Repository: https://github.com/rdegon/rdegon-siem-solution

## Overview

#### Purpose and Entry Points
Rdegon Sentinel is a homelab SIEM/SOC platform: it collects endpoint/network/security telemetry, processes and correlates events, stores/query-serves security data, and exposes an operator control plane. Primary runtime entry points are the FastAPI application at `services/web/main.py`, ingest server at `services/ingest/app.py`, worker processes such as `services/normalizer/worker.py` and `services/stream_corr/worker.py`, and deployment orchestration under `deploy/`.

#### Telemetry Collection and Ingest Boundary
Collectors forward syslog, Windows events, host-runtime data, network sensor output, and vulnerability/security telemetry into the ingest fabric. `services/ingest/syslog_server.py` implements network intake; `services/ingest/runtime_state.py` tracks ingest state; `windows-event-agent/src/Rdegon.WindowsEventAgent/WindowsEventCollectorService.cs` supplies a managed Windows collector with local spool support. Host/system forwarding configuration and sensor deployment are represented by `deploy/common/90-siem-forward.conf`, `deploy/security_sensor_forwarder.py`, and `deploy/windows-agent/install-windows-event-agent.ps1`.

#### Stream Processing Pipeline
The core asynchronous path is ingest → transport → filtering/normalization → correlation → persistence/alerts. Kafka is the material transport boundary, provisioned through `deploy/kafka_topic_bootstrap.py` and node services such as `deploy/vm1/siem-kafka.service`. `services/filter/worker.py` applies filtering, `services/normalizer/normalizer_core.py` converts source events into shared representations, and `services/writer/worker.py` persists processed records. Redis-backed operational/stream state is encapsulated by `services/stream_state.py` and `services/redis_runtime.py`.

#### Detection, Correlation, and Incident Delivery
Streaming correlation evaluates versioned rule packs through `services/stream_corr/rules.py` and `services/stream_corr/worker.py`; packs are source-controlled JSON, for example `correlation_rule_packs/siem_detection_pack_v1.json` and `correlation_rule_packs/host_runtime_policy_v1.json`. Batch/detection schemas and seeds live in `sql/10_detection_rule_catalog.sql` and `sql/13_batch_corr_seed.sql`. Alert aggregation is a distinct worker (`services/alert_agg/worker.py`), while operational incident notification/AI integrations are handled by `services/incident_telegram_bot.py` and `services/web/app/incident_ai_runtime.py`.

#### Storage, Query, and Control Plane
ClickHouse is the primary event/query store, with schema evolution under `sql/11_event_schema_enrichment.sql` and `sql/20_event_hunting.sql`; production safeguards are automated in `deploy/clickhouse_hot_storage_guard.py`. The FastAPI control plane exposes APIs, console routes, and legacy server-rendered pages: `services/web/main.py`, `services/web/app/routes/console_router_registry.py`, and query modules such as `services/web/app/query/events.py`. It also manages operational domains—assets, access, topology, lifecycle, vulnerability exposure, and response workflows—rather than merely displaying events.

#### Operator UI and Identity Boundary
The React operator SPA enters at `frontend-react/src/main.tsx` and is composed around `frontend-react/src/sentinel/App.tsx`, with API access isolated in `frontend-react/src/sentinel/runtime/api.ts`. It is served alongside the web control plane under `/app/*`; server-rendered templates remain under `services/web/app/templates/`. Authentication/authorization integrates OIDC and Keycloak through `services/web/app/oidc_runtime.py` and `services/web/app/keycloak_admin_runtime.py`, separating external identity administration from console-facing access operations.

#### Infrastructure and Operations
Deployment is VM-oriented and systemd-managed: VM1 ingest, VM2 processing, VM3 writers/storage integration, VM4 control-plane services, and VM5 transport/standby roles are codified in scripts such as `deploy/vm1_ingest_fabric_deploy.py`, `deploy/vm2_processing_remote_deploy.py`, and `deploy/vm4_enterprise_foundation_deploy.py`. External/infrastructure integrations include Proxmox, Vault, Keycloak, Kafka, Redis, ClickHouse, Mongo/Postgres-related control-plane cutovers, MinIO, MISP, Greenbone/OpenVAS, Arkime, Zeek, Suricata, Velociraptor, OPNsense, Telegram, and VPN/VLESS access services. Deployment scripts and systemd units form the production control boundary; runtime secrets remain external to the repository.

Main entry points: [`services/stream_corr/worker.py`](https://github.com/rdegon/rdegon-siem-solution/blob/main/services/stream_corr/worker.py) (Streaming correlation), [`services/web/main.py`](https://github.com/rdegon/rdegon-siem-solution/blob/main/services/web/main.py) (Web control plane), [`services/ingest/app.py`](https://github.com/rdegon/rdegon-siem-solution/blob/main/services/ingest/app.py) (Ingest service), [`deploy/vm1_ingest_fabric_deploy.py`](https://github.com/rdegon/rdegon-siem-solution/blob/main/deploy/vm1_ingest_fabric_deploy.py) (VM systemd deployment)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Collection & Ingest"]
  node_collectors{{"Endpoint &amp; sensor collectors<br/>telemetry sources"}}
  node_windows_agent["Windows event agent<br/>managed collector"]
  node_ingest["Ingest service<br/>FastAPI ingest<br/>[app.py]"]
end

subgraph group_pipeline["Stream Pipeline"]
  node_kafka["Kafka transport<br/>event bus"]
  node_filter["Filter worker<br/>stream worker<br/>[worker.py]"]
  node_normalizer["Normalization<br/>stream worker<br/>[normalizer_core.py]"]
  node_stream_state[("Redis stream state<br/>operational state<br/>[stream_state.py]")]
  node_correlation["Streaming correlation<br/>stream worker<br/>[worker.py]"]
  node_rule_packs["Versioned rule packs<br/>detection rules"]
  node_writer["Writer worker<br/>persistence worker<br/>[worker.py]"]
  node_alert_aggregation["Alert aggregation<br/>stream worker<br/>[worker.py]"]
  node_clickhouse[("ClickHouse<br/>event query store")]
end

subgraph group_control["Control Plane"]
  node_web_api["Web control plane<br/>FastAPI application<br/>[main.py]"]
  node_query_api["Event query modules<br/>query API<br/>[events.py]"]
  node_operator_spa["Operator SPA<br/>React application<br/>[App.tsx]"]
  node_identity["OIDC &amp; Keycloak<br/>identity integration<br/>[oidc_runtime.py]"]
  node_incident_delivery["Incident delivery &amp; AI<br/>notification integration"]
end

subgraph group_operations["Deployment &amp; External Systems"]
  node_vm_deployment["VM systemd deployment<br/>production orchestration"]
  node_external_services{{"External security services<br/>integrations"}}
end

node_collectors -->|"forwards telemetry"| node_ingest
node_windows_agent -->|"forwards events"| node_ingest
node_ingest -->|"publishes"| node_kafka
node_kafka -->|"consumes"| node_filter
node_filter -->|"filtered events"| node_normalizer
node_normalizer -->|"normalized events"| node_correlation
node_normalizer -->|"processed records"| node_writer
node_correlation -.->|"evaluates"| node_rule_packs
node_correlation -->|"reads and updates"| node_stream_state
node_correlation -->|"detections"| node_alert_aggregation
node_writer -->|"persists"| node_clickhouse
node_alert_aggregation -->|"stores alerts"| node_clickhouse
node_alert_aggregation -->|"incident notifications"| node_incident_delivery
node_web_api -->|"uses"| node_query_api
node_query_api -->|"hunts and queries"| node_clickhouse
node_operator_spa -->|"API access"| node_web_api
node_web_api -->|"authenticates and administers"| node_identity
node_vm_deployment -.->|"deploys"| node_ingest
node_vm_deployment -.->|"provisions"| node_kafka
node_vm_deployment -.->|"deploys"| node_web_api
node_external_services -.->|"telemetry integrations"| node_collectors
node_external_services -.->|"notification integrations"| node_incident_delivery

click node_collectors "https://github.com/rdegon/rdegon-siem-solution/blob/main/deploy/security_sensor_forwarder.py"
click node_windows_agent "https://github.com/rdegon/rdegon-siem-solution/blob/main/windows-event-agent/src/Rdegon.WindowsEventAgent/WindowsEventCollectorService.cs"
click node_ingest "https://github.com/rdegon/rdegon-siem-solution/blob/main/services/ingest/app.py"
click node_kafka "https://github.com/rdegon/rdegon-siem-solution/blob/main/deploy/kafka_topic_bootstrap.py"
click node_filter "https://github.com/rdegon/rdegon-siem-solution/blob/main/services/filter/worker.py"
click node_normalizer "https://github.com/rdegon/rdegon-siem-solution/blob/main/services/normalizer/normalizer_core.py"
click node_stream_state "https://github.com/rdegon/rdegon-siem-solution/blob/main/services/stream_state.py"
click node_correlation "https://github.com/rdegon/rdegon-siem-solution/blob/main/services/stream_corr/worker.py"
click node_rule_packs "https://github.com/rdegon/rdegon-siem-solution/blob/main/correlation_rule_packs/siem_detection_pack_v1.json"
click node_writer "https://github.com/rdegon/rdegon-siem-solution/blob/main/services/writer/worker.py"
click node_alert_aggregation "https://github.com/rdegon/rdegon-siem-solution/blob/main/services/alert_agg/worker.py"
click node_clickhouse "https://github.com/rdegon/rdegon-siem-solution/blob/main/sql/11_event_schema_enrichment.sql"
click node_web_api "https://github.com/rdegon/rdegon-siem-solution/blob/main/services/web/main.py"
click node_query_api "https://github.com/rdegon/rdegon-siem-solution/blob/main/services/web/app/query/events.py"
click node_operator_spa "https://github.com/rdegon/rdegon-siem-solution/blob/main/frontend-react/src/sentinel/App.tsx"
click node_identity "https://github.com/rdegon/rdegon-siem-solution/blob/main/services/web/app/oidc_runtime.py"
click node_incident_delivery "https://github.com/rdegon/rdegon-siem-solution/blob/main/services/incident_telegram_bot.py"
click node_vm_deployment "https://github.com/rdegon/rdegon-siem-solution/blob/main/deploy/vm1_ingest_fabric_deploy.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_collectors,node_windows_agent,node_ingest toneBlue
class node_kafka,node_filter,node_normalizer,node_stream_state,node_correlation,node_rule_packs,node_writer,node_alert_aggregation,node_clickhouse toneAmber
class node_web_api,node_query_api,node_operator_spa,node_identity,node_incident_delivery toneMint
class node_vm_deployment,node_external_services toneRose
```

## Components

### Collection & Ingest

- [Endpoint & sensor collectors](https://github.com/rdegon/rdegon-siem-solution/blob/main/deploy/security_sensor_forwarder.py): telemetry sources · Syslog, Windows, host runtime, network, and security telemetry. · `deploy/security_sensor_forwarder.py`
- [Windows event agent](https://github.com/rdegon/rdegon-siem-solution/blob/main/windows-event-agent/src/Rdegon.WindowsEventAgent/WindowsEventCollectorService.cs): managed collector · Collects Windows events with local spool support. · `windows-event-agent/src/Rdegon.WindowsEventAgent/WindowsEventCollectorService.cs`
- [Ingest service](https://github.com/rdegon/rdegon-siem-solution/blob/main/services/ingest/app.py): FastAPI ingest · Network telemetry intake and ingest-state boundary. · `services/ingest/app.py`

### Stream Pipeline

- [Kafka transport](https://github.com/rdegon/rdegon-siem-solution/blob/main/deploy/kafka_topic_bootstrap.py): event bus · Material asynchronous transport boundary. · `deploy/kafka_topic_bootstrap.py`
- [Filter worker](https://github.com/rdegon/rdegon-siem-solution/blob/main/services/filter/worker.py): stream worker · `services/filter/worker.py`
- [Normalization](https://github.com/rdegon/rdegon-siem-solution/blob/main/services/normalizer/normalizer_core.py): stream worker · Converts source events to shared representations. · `services/normalizer/normalizer_core.py`
- [Redis stream state](https://github.com/rdegon/rdegon-siem-solution/blob/main/services/stream_state.py): operational state · Redis-backed stream and operational state. · `services/stream_state.py`
- [Streaming correlation](https://github.com/rdegon/rdegon-siem-solution/blob/main/services/stream_corr/worker.py): stream worker · Evaluates detection rules against normalized events. · `services/stream_corr/worker.py`
- [Versioned rule packs](https://github.com/rdegon/rdegon-siem-solution/blob/main/correlation_rule_packs/siem_detection_pack_v1.json): detection rules · `correlation_rule_packs/siem_detection_pack_v1.json`
- [Writer worker](https://github.com/rdegon/rdegon-siem-solution/blob/main/services/writer/worker.py): persistence worker · `services/writer/worker.py`
- [Alert aggregation](https://github.com/rdegon/rdegon-siem-solution/blob/main/services/alert_agg/worker.py): stream worker · `services/alert_agg/worker.py`
- [ClickHouse](https://github.com/rdegon/rdegon-siem-solution/blob/main/sql/11_event_schema_enrichment.sql): event query store · Primary security event and hunting store. · `sql/11_event_schema_enrichment.sql`

### Control Plane

- [Web control plane](https://github.com/rdegon/rdegon-siem-solution/blob/main/services/web/main.py): FastAPI application · APIs and operational domains for assets, access, topology, lifecycle, exposure, and response. · `services/web/main.py`
- [Event query modules](https://github.com/rdegon/rdegon-siem-solution/blob/main/services/web/app/query/events.py): query API · `services/web/app/query/events.py`
- [Operator SPA](https://github.com/rdegon/rdegon-siem-solution/blob/main/frontend-react/src/sentinel/App.tsx): React application · Served under /app/* alongside the control plane. · `frontend-react/src/sentinel/App.tsx`
- [OIDC & Keycloak](https://github.com/rdegon/rdegon-siem-solution/blob/main/services/web/app/oidc_runtime.py): identity integration · Separates console access from external identity administration. · `services/web/app/oidc_runtime.py`
- [Incident delivery & AI](https://github.com/rdegon/rdegon-siem-solution/blob/main/services/incident_telegram_bot.py): notification integration · `services/incident_telegram_bot.py`

### Deployment & External Systems

- [VM systemd deployment](https://github.com/rdegon/rdegon-siem-solution/blob/main/deploy/vm1_ingest_fabric_deploy.py): production orchestration · Deploys ingest, processing, writer, control-plane, and transport roles. · `deploy/vm1_ingest_fabric_deploy.py`
- **External security services**: integrations · Sensors, vulnerability, threat intelligence, storage, identity, and access services.

## Connections

- Endpoint & sensor collectors → Ingest service: forwards telemetry
- Windows event agent → Ingest service: forwards events
- Ingest service → Kafka transport: publishes
- Kafka transport → Filter worker: consumes
- Filter worker → Normalization: filtered events
- Normalization → Streaming correlation: normalized events
- Normalization → Writer worker: processed records
- Streaming correlation → Versioned rule packs: evaluates
- Streaming correlation → Redis stream state: reads and updates
- Streaming correlation → Alert aggregation: detections
- Writer worker → ClickHouse: persists
- Alert aggregation → ClickHouse: stores alerts
- Alert aggregation → Incident delivery & AI: incident notifications
- Web control plane → Event query modules: uses
- Event query modules → ClickHouse: hunts and queries
- Operator SPA → Web control plane: API access
- Web control plane → OIDC & Keycloak: authenticates and administers
- VM systemd deployment → Ingest service: deploys
- VM systemd deployment → Kafka transport: provisions
- VM systemd deployment → Web control plane: deploys
- External security services → Endpoint & sensor collectors: telemetry integrations
- External security services → Incident delivery & AI: notification integrations