# ahmed-sobhi-ali/custom-siem-pipeline-with-ai-powered-detection architecture

> Architecture diagram of the GitHub repository ahmed-sobhi-ali/custom-siem-pipeline-with-ai-powered-detection, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ahmed-sobhi-ali/custom-siem-pipeline-with-ai-powered-detection
- Repository: https://github.com/ahmed-sobhi-ali/custom-siem-pipeline-with-ai-powered-detection

## Overview

##### Project Type and Purpose
- This is a full-stack, modular SIEM (Security Information and Event Management) pipeline written in Python.
- Purpose: collect system logs in real time, parse and structure them, extract ML-ready features, detect anomalies via AI models, and send alerts through multiple channels.

a. Log Forwarder
- File: log_forwarder.py
- Reads from systemd.journal
- Batches logs, retries on failure
- Sends via HTTP/TCP/UDP/Syslog
b. Log Receiver (Collector)
- File: log_collector.py
- Multithreaded HTTP/TCP/UDP server
- Persists raw `.log` files per host/date
c. Log Parser
- File: log_parser.py
- Tails raw `.log` files
- Regex-based syslog parsing → JSON
- Timestamp normalization
d. Feature Extractor
- File: features_extraction.py
- Reads `logs_parsed.json` → writes `logs_features.jsonl`
- Generates temporal, categorical, text features
e. Anomaly Detector
- File: anomaly_detection.py
- Supports Isolation Forest, LOF, One-Class SVM
- Scales/numerical + TF-IDF/text vectors → concatenated
- Modes: initial training vs. batch prediction
f. Alert Manager
- Integrated in anomaly_detection.py (or separate module)
- Channels: console, webhook, syslog, local file
- Formats alerts with log ID, score, algorithm, raw line
g. Real-Time Orchestrator
- Class: RealTimeAnomalyDetectionSystem
- Uses `watchdog` to monitor feature file
- Triggers detection batches by size or timeout
- Ties all components into a continuous pipeline

##### Component Interactions and Data Flow
1. kernel logs → Log Forwarder → over network
2. Log Receiver → raw `.log` files
3. Log Parser tails raw files → emits structured JSON
4. Feature Extractor ingests JSON → produces feature records
5. Real-Time Orchestrator watches feature file → batches records
6. Anomaly Detector trains/predicts → outputs anomaly scores
7. Alert Manager sends alerts to configured endpoints

##### Architectural Patterns & Design Principles
- Modular, pipeline architecture: each stage is a standalone process/script.
- Microservices-style communication over standard protocols (HTTP/TCP/UDP).
- Clear separation of concerns (collection, parsing, feature engineering, detection, alerting).
- Extensibility: swap in new parsing rules, add ML algorithms, plug new alert channels.
- Real-time streaming vs. batch processing (hybrid via file watching).

##### Technologies, Frameworks & Libraries
- Python 3.x
- systemd Python bindings (journal access)
- HTTP server (Flask or built-in), sockets for TCP/UDP
- watchfiles/watchdog for file monitoring
- Regex, JSON for parsing/serialization
- scikit-learn for Isolation Forest, LOF, One-Class SVM
- TF-IDF vectorizer from scikit-learn
- Threading or asyncio for concurrency

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion"
        direction TB
        LF["Log Forwarder"]:::ingestion
        LR["Log Receiver"]:::ingestion
    end

    %% Data Transformation Layer
    subgraph "Data Transformation"
        direction TB
        LP["Log Parser"]:::transform
        FE["Feature Extractor"]:::transform
    end

    %% AI/ML Processing Layer
    subgraph "AI/ML Processing"
        direction TB
        AD["Anomaly Detector"]:::ml
        AM["Alert Manager"]:::alert
    end

    %% Orchestration Layer
    subgraph "Orchestration"
        direction TB
        OR["Real-Time Orchestrator"]:::orchestrator
    end

    %% Data Flow
    LF -->|"HTTP/TCP/UDP/Syslog"| LR
    LR -->|"raw .log files (fs)"| LP
    LP -->|"logs_parsed.json (fs)"| FE
    FE -->|"logs_features.jsonl (fs)"| OR
    OR -->|"batch trigger"| AD
    AD -->|"anomaly scores"| AM

    %% Legend
    subgraph "Legend"
        direction TB
        L1["Data Ingestion"]:::ingestion
        L2["Data Transformation"]:::transform
        L3["AI/ML Processing"]:::ml
        L4["Alerting"]:::alert
        L5["Orchestration"]:::orchestrator
    end

    %% Click Events
    click LF "https://github.com/ahmed-sobhi-ali/custom-siem-pipeline-with-ai-powered-detection/blob/main/log_forwarder.py"
    click LR "https://github.com/ahmed-sobhi-ali/custom-siem-pipeline-with-ai-powered-detection/blob/main/log_collector.py"
    click LP "https://github.com/ahmed-sobhi-ali/custom-siem-pipeline-with-ai-powered-detection/blob/main/log_parser.py"
    click FE "https://github.com/ahmed-sobhi-ali/custom-siem-pipeline-with-ai-powered-detection/blob/main/features_extraction.py"
    click AD "https://github.com/ahmed-sobhi-ali/custom-siem-pipeline-with-ai-powered-detection/blob/main/anomaly_detection.py"
    click AM "https://github.com/ahmed-sobhi-ali/custom-siem-pipeline-with-ai-powered-detection/blob/main/anomaly_detection.py"
    click OR "https://github.com/ahmed-sobhi-ali/custom-siem-pipeline-with-ai-powered-detection/blob/main/anomaly_detection.py"

    %% Styles
    classDef ingestion fill:#bbdefb,stroke:#1e88e5;
    classDef transform fill:#c8e6c9,stroke:#388e3c;
    classDef ml fill:#ffe0b2,stroke:#f57c00;
    classDef alert fill:#ffcdd2,stroke:#d32f2f;
    classDef orchestrator fill:#e1bee7,stroke:#8e24aa;
```
