rdegon/rdegon-siem-solution

Purpose and Entry Points

Rdegon Sentinel is a homelab SIEM/SOC platform: it collects endpoint/network/security telemetry, processes and correlates events, stores/query-serves security data, and exposes an operator control plane. Primary runtime entry points are the FastAPI application at services/web/main.py, ingest server at services/ingest/app.py, worker processes such as services/normalizer/worker.py and services/stream_corr/worker.py, and deployment orchestration under deploy/.

Telemetry Collection and Ingest Boundary

Collectors forward syslog, Windows events, host-runtime data, network sensor output, and vulnerability/security telemetry into the ingest fabric. services/ingest/syslog_server.py implements network intake; services/ingest/runtime_state.py tracks ingest state; windows-event-agent/src/Rdegon.WindowsEventAgent/WindowsEventCollectorService.cs supplies a managed Windows collector with local spool support. Host/system forwarding configuration and sensor deployment are represented by deploy/common/90-siem-forward.conf, deploy/security_sensor_forwarder.py, and deploy/windows-agent/install-windows-event-agent.ps1.

Stream Processing Pipeline

The core asynchronous path is ingest → transport → filtering/normalization → correlation → persistence/alerts. Kafka is the material transport boundary, provisioned through deploy/kafka_topic_bootstrap.py and node services such as deploy/vm1/siem-kafka.service. services/filter/worker.py applies filtering, services/normalizer/normalizer_core.py converts source events into shared representations, and services/writer/worker.py persists processed records. Redis-backed operational/stream state is encapsulated by services/stream_state.py and services/redis_runtime.py.

Detection, Correlation, and Incident Delivery

Streaming correlation evaluates versioned rule packs through services/stream_corr/rules.py and services/stream_corr/worker.py; packs are source-controlled JSON, for example correlation_rule_packs/siem_detection_pack_v1.json and correlation_rule_packs/host_runtime_policy_v1.json. Batch/detection schemas and seeds live in sql/10_detection_rule_catalog.sql and sql/13_batch_corr_seed.sql. Alert aggregation is a distinct worker (services/alert_agg/worker.py), while operational incident notification/AI integrations are handled by services/incident_telegram_bot.py and services/web/app/incident_ai_runtime.py.

Storage, Query, and Control Plane

ClickHouse is the primary event/query store, with schema evolution under sql/11_event_schema_enrichment.sql and sql/20_event_hunting.sql; production safeguards are automated in deploy/clickhouse_hot_storage_guard.py. The FastAPI control plane exposes APIs, console routes, and legacy server-rendered pages: services/web/main.py, services/web/app/routes/console_router_registry.py, and query modules such as services/web/app/query/events.py. It also manages operational domains—assets, access, topology, lifecycle, vulnerability exposure, and response workflows—rather than merely displaying events.

Operator UI and Identity Boundary

The React operator SPA enters at frontend-react/src/main.tsx and is composed around frontend-react/src/sentinel/App.tsx, with API access isolated in frontend-react/src/sentinel/runtime/api.ts. It is served alongside the web control plane under /app/*; server-rendered templates remain under services/web/app/templates/. Authentication/authorization integrates OIDC and Keycloak through services/web/app/oidc_runtime.py and services/web/app/keycloak_admin_runtime.py, separating external identity administration from console-facing access operations.

Infrastructure and Operations

Deployment is VM-oriented and systemd-managed: VM1 ingest, VM2 processing, VM3 writers/storage integration, VM4 control-plane services, and VM5 transport/standby roles are codified in scripts such as deploy/vm1_ingest_fabric_deploy.py, deploy/vm2_processing_remote_deploy.py, and deploy/vm4_enterprise_foundation_deploy.py. External/infrastructure integrations include Proxmox, Vault, Keycloak, Kafka, Redis, ClickHouse, Mongo/Postgres-related control-plane cutovers, MinIO, MISP, Greenbone/OpenVAS, Arkime, Zeek, Suricata, Velociraptor, OPNsense, Telegram, and VPN/VLESS access services. Deployment scripts and systemd units form the production control boundary; runtime secrets remain external to the repository.