# sxjalxo/siemlite architecture

> Architecture diagram of the GitHub repository sxjalxo/siemlite, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sxjalxo/siemlite
- Repository: https://github.com/sxjalxo/siemlite

## Overview

1. **Repository purpose and shape**
   - SIEM-Lite is a single Python application that combines two primary analysis paths: **active web/network scanning** and **passive log analysis**.
   - The architecture is centered around a **unified CLI entrypoint** in `main.py`, which orchestrates scanning, log ingestion, correlation, alerting, and reporting.

2. **Top-level orchestration**
   - `main.py` is the system coordinator. It accepts user intent such as “scan”, “analyze”, or both, then drives the relevant subsystems and combines their outputs into a unified result set.
   - The same orchestration layer also performs **cross-module correlation**, turning individual findings into higher-confidence threat detections.

3. **Active scanning subsystem**
   - `core/scanner.py` handles direct interaction with a target host over HTTP and TCP.
   - Its responsibilities include SQL injection probing, reflected XSS checks, security-header inspection, and port scanning.
   - This subsystem is outbound-facing: it generates evidence from live target interaction rather than from stored data.

4. **Log analysis subsystem**
   - `core/log_analyzer.py` processes access/security logs such as the samples in `data/`.
   - It detects brute-force patterns, suspicious IP frequency, 404 scanning, malicious URIs, and web-shell access.
   - This is the passive analysis path and is the main consumer of file-based evidence.

5. **Shared detection and scoring boundary**
   - `core/detection_engine.py` and the correlation logic in `main.py` represent the evaluation layer that turns raw module results into threat judgments.
   - Findings from scanner and log analyzer are combined by severity/scoring rules to produce higher-level incidents such as “open SSH port + brute force attempts” or “SQLi vulnerability + SQLi attempts in logs”.

6. **Enrichment and advanced analytics modules**
   - The `modules/` package adds contextual enrichment rather than primary detection.
   - `geo_ip.py` provides geolocation and impossible-travel style checks.
   - `mitre_mapping.py` maps detections to MITRE ATT&CK techniques.
   - `threat_intel.py` checks IP reputation via AbuseIPDB and a local blacklist.
   - `anomaly_detection.py` adds statistical spike detection, likely over request or event volumes.

7. **Ingestion and real-time monitoring**
   - `ingestion/realtime_monitor.py` is the event-driven input path for live log monitoring.
   - It uses filesystem watching to detect log updates and feed new entries into the analysis pipeline.
   - This creates a second ingestion mode alongside one-shot file analysis.

8. **Extensibility via custom rules**
   - `core/rule_loader.py` and the `rules/` directory implement a plugin boundary for user-defined detections.
   - Custom rule files expose a `detect(entries)` interface, letting the system load additional rules without modifying the core analyzer.
   - `rules/example_rule.py` serves as the template for this extension point.

9. **Persistence and state**
   - `core/database.py` stores results in SQLite under `db/siem_lite.db`.
   - The database is used for durable retention of logs, alerts, correlations, and statistics rather than just in-memory reporting.
   - This is the main state boundary in the system: analysis may be transient, but selected outputs are persisted.

10. **Alerting and output channels**
   - `core/alerting.py` fans out notable findings to multiple destinations: log files, JSON output, email, and desktop notifications.
   - `output/` contains generated artifacts such as alert logs and JSON reports.
   - This subsystem is downstream of detection and correlation, acting as the notification/output layer.

11. **Presentation layer**
   - `ui/dashboard.py` provides an interactive CLI dashboard built with `rich`.
   - It likely reads from computed results and/or the SQLite store to present severity breakdowns, trends, and recent findings.
   - This is a presentation-only boundary; it should not be the place where detections are made.

12. **Inputs, assets, and external dependencies**
   - `data/sample_access.log` and `samples/` provide test inputs for log analysis.
   - `data/blacklist.json` supports local threat-intel lookups.
   - External services and assets that materially affect behavior are: **AbuseIPDB** for reputation checks, **MaxMind GeoLite2** for geolocation, **SMTP** for email alerts, and the local SQLite file for persistence.
   - Tooling/runtime-wise, the project is a Python 3 application using `requests`, `BeautifulSoup4`, `watchdog`, `rich`, `geoip2`, and standard libraries like `socket`, `re`, `argparse`, and `sqlite3`.

13. **End-to-end data flow**
   - Typical flow: CLI input → scanner and/or log analyzer → enrichment modules and custom rules → correlation/scoring → alerting/reporting → optional persistence in SQLite and dashboard display.
   - The key architectural distinction is between **evidence generation** (scan/log ingest), **evidence enrichment** (geo/IP/mitre/threat intel/anomaly), and **decision/output** (correlation, alerts, reports, dashboard).

Main entry points: [`core/detection_engine.py`](https://github.com/sxjalxo/siemlite/blob/main/core/detection_engine.py) (Detection Engine), [`main.py`](https://github.com/sxjalxo/siemlite/blob/main/main.py) (main.py), [`core/log_analyzer.py`](https://github.com/sxjalxo/siemlite/blob/main/core/log_analyzer.py) (Log Analyzer), [`ingestion/log_ingestor.py`](https://github.com/sxjalxo/siemlite/blob/main/ingestion/log_ingestor.py) (Log Ingestor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_orchestration["Orchestration"]
  node_main_py["main.py<br/>CLI orchestrator<br/>[main.py]"]
end

subgraph group_grp_analysis["Analysis"]
  node_scanner["Scanner<br/>active scan<br/>[scanner.py]"]
  node_log_analyzer["Log Analyzer<br/>passive analysis<br/>[log_analyzer.py]"]
  node_realtime_monitor["Realtime Monitor<br/>log watcher"]
  node_log_ingestor["Log Ingestor<br/>ingestion adapter<br/>[log_ingestor.py]"]
  node_rule_loader["Rule Loader<br/>plugin loader<br/>[rule_loader.py]"]
  node_example_rule["Example Rule<br/>custom rule<br/>[example_rule.py]"]
  node_detection_engine["Detection Engine<br/>scoring layer"]
end

subgraph group_grp_enrichment["Enrichment"]
  node_geo_ip["Geo IP<br/>enrichment<br/>[geo_ip.py]"]
  node_threat_intel["Threat Intel<br/>reputation lookup<br/>[threat_intel.py]"]
  node_mitre_mapping["MITRE Map<br/>tactic mapping<br/>[mitre_mapping.py]"]
  node_anomaly_detection["Anomaly Detect<br/>statistical analytics"]
end

subgraph group_grp_output["Output & State"]
  node_alerting["Alerting<br/>notification fanout<br/>[alerting.py]"]
  node_database[("SQLite Store<br/>persistence<br/>[database.py]")]
  node_report["Report<br/>[report.py]"]
  node_dashboard["Dashboard<br/>CLI UI<br/>[dashboard.py]"]
end

subgraph group_grp_inputs["Inputs"]
  node_sample_log["Sample Log<br/>input data<br/>[sample_access.log]"]
  node_blacklist["Blacklist<br/>reference data"]
end

node_main_py -->|"scan"| node_scanner
node_main_py -->|"analyze"| node_log_analyzer
node_main_py -->|"live mode"| node_realtime_monitor
node_realtime_monitor -->|"events"| node_log_ingestor
node_log_ingestor -->|"entries"| node_log_analyzer
node_log_analyzer -->|"rules"| node_rule_loader
node_rule_loader -->|"loads"| node_example_rule
node_scanner -->|"findings"| node_detection_engine
node_log_analyzer -->|"findings"| node_detection_engine
node_main_py -->|"correlate"| node_detection_engine
node_detection_engine -->|"enrich"| node_geo_ip
node_detection_engine -->|"enrich"| node_threat_intel
node_detection_engine -->|"map"| node_mitre_mapping
node_detection_engine -->|"score"| node_anomaly_detection
node_threat_intel -->|"local lookup"| node_blacklist
node_detection_engine -->|"alerts"| node_alerting
node_detection_engine -->|"persist"| node_database
node_alerting -->|"output"| node_report
node_database -->|"read"| node_dashboard
node_main_py -->|"summarize"| node_report
node_sample_log -->|"input"| node_log_ingestor

click node_main_py "https://github.com/sxjalxo/siemlite/blob/main/main.py"
click node_scanner "https://github.com/sxjalxo/siemlite/blob/main/core/scanner.py"
click node_log_analyzer "https://github.com/sxjalxo/siemlite/blob/main/core/log_analyzer.py"
click node_realtime_monitor "https://github.com/sxjalxo/siemlite/blob/main/ingestion/realtime_monitor.py"
click node_log_ingestor "https://github.com/sxjalxo/siemlite/blob/main/ingestion/log_ingestor.py"
click node_rule_loader "https://github.com/sxjalxo/siemlite/blob/main/core/rule_loader.py"
click node_example_rule "https://github.com/sxjalxo/siemlite/blob/main/rules/example_rule.py"
click node_detection_engine "https://github.com/sxjalxo/siemlite/blob/main/core/detection_engine.py"
click node_geo_ip "https://github.com/sxjalxo/siemlite/blob/main/modules/geo_ip.py"
click node_threat_intel "https://github.com/sxjalxo/siemlite/blob/main/modules/threat_intel.py"
click node_mitre_mapping "https://github.com/sxjalxo/siemlite/blob/main/modules/mitre_mapping.py"
click node_anomaly_detection "https://github.com/sxjalxo/siemlite/blob/main/modules/anomaly_detection.py"
click node_alerting "https://github.com/sxjalxo/siemlite/blob/main/core/alerting.py"
click node_database "https://github.com/sxjalxo/siemlite/blob/main/core/database.py"
click node_report "https://github.com/sxjalxo/siemlite/blob/main/core/report.py"
click node_dashboard "https://github.com/sxjalxo/siemlite/blob/main/ui/dashboard.py"
click node_sample_log "https://github.com/sxjalxo/siemlite/blob/main/data/sample_access.log"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_py toneBlue
class node_scanner,node_log_analyzer,node_realtime_monitor,node_log_ingestor,node_rule_loader,node_example_rule,node_detection_engine toneAmber
class node_geo_ip,node_threat_intel,node_mitre_mapping,node_anomaly_detection toneMint
class node_alerting,node_database,node_report,node_dashboard toneRose
class node_sample_log,node_blacklist toneIndigo
```

## Components

### Orchestration

Entry point and coordination layer

- [main.py](https://github.com/sxjalxo/siemlite/blob/main/main.py): CLI orchestrator · Unified entrypoint that routes scan/analyze flows and correlates findings · `main.py`

### Analysis

Active scanning, passive log analysis, and rule loading

- [Scanner](https://github.com/sxjalxo/siemlite/blob/main/core/scanner.py): active scan · HTTP/TCP probing for SQLi, XSS, headers, and ports · `core/scanner.py`
- [Log Analyzer](https://github.com/sxjalxo/siemlite/blob/main/core/log_analyzer.py): passive analysis · Detects brute force, suspicious IPs, 404 scanning, malicious URIs, and web shells · `core/log_analyzer.py`
- [Realtime Monitor](https://github.com/sxjalxo/siemlite/blob/main/ingestion/realtime_monitor.py): log watcher · Watches filesystem changes and feeds new log entries into analysis · `ingestion/realtime_monitor.py`
- [Log Ingestor](https://github.com/sxjalxo/siemlite/blob/main/ingestion/log_ingestor.py): ingestion adapter · Bridges raw log sources into the analysis pipeline · `ingestion/log_ingestor.py`
- [Rule Loader](https://github.com/sxjalxo/siemlite/blob/main/core/rule_loader.py): plugin loader · Loads custom detection rules from the rules package · `core/rule_loader.py`
- [Example Rule](https://github.com/sxjalxo/siemlite/blob/main/rules/example_rule.py): custom rule · Template rule implementing the detect(entries) extension point · `rules/example_rule.py`
- [Detection Engine](https://github.com/sxjalxo/siemlite/blob/main/core/detection_engine.py): scoring layer · Evaluates findings and applies severity/scoring rules · `core/detection_engine.py`

### Enrichment

Contextual analytics and threat mapping

- [Geo IP](https://github.com/sxjalxo/siemlite/blob/main/modules/geo_ip.py): enrichment · Geolocation and travel-style checks for source IP context · `modules/geo_ip.py`
- [Threat Intel](https://github.com/sxjalxo/siemlite/blob/main/modules/threat_intel.py): reputation lookup · Checks IP reputation against AbuseIPDB and local blacklist · `modules/threat_intel.py`
- [MITRE Map](https://github.com/sxjalxo/siemlite/blob/main/modules/mitre_mapping.py): tactic mapping · Maps detections to MITRE ATT&CK techniques · `modules/mitre_mapping.py`
- [Anomaly Detect](https://github.com/sxjalxo/siemlite/blob/main/modules/anomaly_detection.py): statistical analytics · Flags spikes and unusual event-volume patterns · `modules/anomaly_detection.py`

### Output & State

Alerting, persistence, reporting, and presentation

- [Alerting](https://github.com/sxjalxo/siemlite/blob/main/core/alerting.py): notification fanout · Sends notable findings to logs, JSON, email, and desktop notifications · `core/alerting.py`
- [SQLite Store](https://github.com/sxjalxo/siemlite/blob/main/core/database.py): persistence · Durable storage for logs, alerts, correlations, and statistics · `core/database.py`
- [Report](https://github.com/sxjalxo/siemlite/blob/main/core/report.py): reporting · Generates summarized output artifacts from analyzed results · `core/report.py`
- [Dashboard](https://github.com/sxjalxo/siemlite/blob/main/ui/dashboard.py): CLI UI · Rich-based interactive presentation of findings and trends · `ui/dashboard.py`

### Inputs

Sample data and local reference assets

- [Sample Log](https://github.com/sxjalxo/siemlite/blob/main/data/sample_access.log): input data · Representative access-log input for passive analysis · `data/sample_access.log`
- **Blacklist**: reference data · Local IP blacklist used by threat-intel checks

## Connections

- main.py → Scanner: scan · Runs active host/network probing
- main.py → Log Analyzer: analyze · Runs passive log analysis
- main.py → Realtime Monitor: live mode · Enables continuous log monitoring
- Realtime Monitor → Log Ingestor: events · Pushes newly observed log content into ingestion
- Log Ingestor → Log Analyzer: entries · Supplies parsed log entries for detection
- Log Analyzer → Rule Loader: rules · Invokes custom detection rules
- Rule Loader → Example Rule: loads · Loads extension rule modules
- Scanner → Detection Engine: findings · Submits scan results for scoring
- Log Analyzer → Detection Engine: findings · Submits log-derived detections for scoring
- main.py → Detection Engine: correlate · Combines cross-source findings into higher-confidence incidents
- Detection Engine → Geo IP: enrich · Adds source-location context
- Detection Engine → Threat Intel: enrich · Adds IP reputation context
- Detection Engine → MITRE Map: map · Annotates detections with ATT&CK techniques
- Detection Engine → Anomaly Detect: score · Applies spike and anomaly context
- Threat Intel → Blacklist: local lookup · Uses local blacklist as a reputation source
- Detection Engine → Alerting: alerts · Sends notable incidents to notification channels
- Detection Engine → SQLite Store: persist · Stores alerts, correlations, and statistics
- Alerting → Report: output · Feeds generated artifacts and summaries
- SQLite Store → Dashboard: read · Provides stored results for presentation
- main.py → Report: summarize · Produces final run reports
- Sample Log → Log Ingestor: input · Representative log source for analysis
