How ahmed-sobhi-ali/custom-siem-pipeline-with-ai-powered-detection works
Project Type and Purpose
• This is a full-stack, modular SIEM (Security Information and Event Management) pipeline written in Python.
• Purpose: collect system logs in real time, parse and structure them, extract ML-ready features, detect anomalies via AI models, and send alerts through multiple channels.
a. Log Forwarder
• File: log_forwarder.py
• Reads from systemd.journal
• Batches logs, retries on failure
• Sends via HTTP/TCP/UDP/Syslog
b. Log Receiver (Collector)
• File: log_collector.py
• Multithreaded HTTP/TCP/UDP server
• Persists raw .log files per host/date
c. Log Parser
• File: log_parser.py
• Tails raw .log files
• Regex-based syslog parsing → JSON
• Timestamp normalization
d. Feature Extractor
• File: features_extraction.py
• Reads logs_parsed.json → writes logs_features.jsonl
• Generates temporal, categorical, text features
e. Anomaly Detector
• File: anomaly_detection.py
• Supports Isolation Forest, LOF, One-Class SVM
• Scales/numerical + TF-IDF/text vectors → concatenated
• Modes: initial training vs. batch prediction
f. Alert Manager
• Integrated in anomaly_detection.py (or separate module)
• Channels: console, webhook, syslog, local file
• Formats alerts with log ID, score, algorithm, raw line
g. Real-Time Orchestrator
• Class: RealTimeAnomalyDetectionSystem
• Uses watchdog to monitor feature file
• Triggers detection batches by size or timeout
• Ties all components into a continuous pipeline
Component Interactions and Data Flow
1. kernel logs → Log Forwarder → over network
2. Log Receiver → raw .log files
3. Log Parser tails raw files → emits structured JSON
4. Feature Extractor ingests JSON → produces feature records
5. Real-Time Orchestrator watches feature file → batches records
6. Anomaly Detector trains/predicts → outputs anomaly scores
7. Alert Manager sends alerts to configured endpoints
Architectural Patterns & Design Principles
• Modular, pipeline architecture: each stage is a standalone process/script.
• Microservices-style communication over standard protocols (HTTP/TCP/UDP).
• Clear separation of concerns (collection, parsing, feature engineering, detection, alerting).
• Extensibility: swap in new parsing rules, add ML algorithms, plug new alert channels.
• Real-time streaming vs. batch processing (hybrid via file watching).
Technologies, Frameworks & Libraries
• Python 3.x
• systemd Python bindings (journal access)
• HTTP server (Flask or built-in), sockets for TCP/UDP
• watchfiles/watchdog for file monitoring
• Regex, JSON for parsing/serialization
• scikit-learn for Isolation Forest, LOF, One-Class SVM
• TF-IDF vectorizer from scikit-learn
• Threading or asyncio for concurrency