How sxjalxo/siemlite works
1. Repository purpose and shape
- SIEM-Lite is a single Python application that combines two primary analysis paths: active web/network scanning and passive log analysis.
- The architecture is centered around a unified CLI entrypoint in main.py, which orchestrates scanning, log ingestion, correlation, alerting, and reporting.
2. Top-level orchestration
- main.py is the system coordinator. It accepts user intent such as “scan”, “analyze”, or both, then drives the relevant subsystems and combines their outputs into a unified result set.
- The same orchestration layer also performs cross-module correlation, turning individual findings into higher-confidence threat detections.
3. Active scanning subsystem
- core/scanner.py handles direct interaction with a target host over HTTP and TCP.
- Its responsibilities include SQL injection probing, reflected XSS checks, security-header inspection, and port scanning.
- This subsystem is outbound-facing: it generates evidence from live target interaction rather than from stored data.
4. Log analysis subsystem
- core/log_analyzer.py processes access/security logs such as the samples in data/.
- It detects brute-force patterns, suspicious IP frequency, 404 scanning, malicious URIs, and web-shell access.
- This is the passive analysis path and is the main consumer of file-based evidence.
5. Shared detection and scoring boundary
- core/detection_engine.py and the correlation logic in main.py represent the evaluation layer that turns raw module results into threat judgments.
- Findings from scanner and log analyzer are combined by severity/scoring rules to produce higher-level incidents such as “open SSH port + brute force attempts” or “SQLi vulnerability + SQLi attempts in logs”.
6. Enrichment and advanced analytics modules
- The modules/ package adds contextual enrichment rather than primary detection.
- geo_ip.py provides geolocation and impossible-travel style checks.
- mitre_mapping.py maps detections to MITRE ATT&CK techniques.
- threat_intel.py checks IP reputation via AbuseIPDB and a local blacklist.
- anomaly_detection.py adds statistical spike detection, likely over request or event volumes.
7. Ingestion and real-time monitoring
- ingestion/realtime_monitor.py is the event-driven input path for live log monitoring.
- It uses filesystem watching to detect log updates and feed new entries into the analysis pipeline.
- This creates a second ingestion mode alongside one-shot file analysis.
8. Extensibility via custom rules
- core/rule_loader.py and the rules/ directory implement a plugin boundary for user-defined detections.
- Custom rule files expose a detect(entries) interface, letting the system load additional rules without modifying the core analyzer.
- rules/example_rule.py serves as the template for this extension point.
9. Persistence and state
- core/database.py stores results in SQLite under db/siem_lite.db.
- The database is used for durable retention of logs, alerts, correlations, and statistics rather than just in-memory reporting.
- This is the main state boundary in the system: analysis may be transient, but selected outputs are persisted.
10. Alerting and output channels
- core/alerting.py fans out notable findings to multiple destinations: log files, JSON output, email, and desktop notifications.
- output/ contains generated artifacts such as alert logs and JSON reports.
- This subsystem is downstream of detection and correlation, acting as the notification/output layer.
11. Presentation layer
- ui/dashboard.py provides an interactive CLI dashboard built with rich.
- It likely reads from computed results and/or the SQLite store to present severity breakdowns, trends, and recent findings.
- This is a presentation-only boundary; it should not be the place where detections are made.
12. Inputs, assets, and external dependencies
- data/sample_access.log and samples/ provide test inputs for log analysis.
- data/blacklist.json supports local threat-intel lookups.
- External services and assets that materially affect behavior are: AbuseIPDB for reputation checks, MaxMind GeoLite2 for geolocation, SMTP for email alerts, and the local SQLite file for persistence.
- Tooling/runtime-wise, the project is a Python 3 application using requests, BeautifulSoup4, watchdog, rich, geoip2, and standard libraries like socket, re, argparse, and sqlite3.
13. End-to-end data flow
- Typical flow: CLI input → scanner and/or log analyzer → enrichment modules and custom rules → correlation/scoring → alerting/reporting → optional persistence in SQLite and dashboard display.
- The key architectural distinction is between evidence generation (scan/log ingest), evidence enrichment (geo/IP/mitre/threat intel/anomaly), and decision/output (correlation, alerts, reports, dashboard).
Main entry points: detection_engine.py Detection Engine · main.py main.py · log_analyzer.py Log Analyzer · log_ingestor.py Log Ingestor
Components 18
Orchestration
Entry point and coordination layer
- main.py · Unified entrypoint that routes scan/analyze flows and correlates findings
main.py
Analysis
Active scanning, passive log analysis, and rule loading
- Scanner · HTTP/TCP probing for SQLi, XSS, headers, and ports
core/scanner.py - Log Analyzer · Detects brute force, suspicious IPs, 404 scanning, malicious URIs, and web shells
core/log_analyzer.py - Realtime Monitor · Watches filesystem changes and feeds new log entries into analysis
ingestion/realtime_monitor.py - Log Ingestor · Bridges raw log sources into the analysis pipeline
ingestion/log_ingestor.py - Rule Loader · Loads custom detection rules from the rules package
core/rule_loader.py - Example Rule · Template rule implementing the detect(entries) extension point
rules/example_rule.py - Detection Engine · Evaluates findings and applies severity/scoring rules
core/detection_engine.py
Enrichment
Contextual analytics and threat mapping
- Geo IP · Geolocation and travel-style checks for source IP context
modules/geo_ip.py - Threat Intel · Checks IP reputation against AbuseIPDB and local blacklist
modules/threat_intel.py - MITRE Map · Maps detections to MITRE ATT&CK techniques
modules/mitre_mapping.py - Anomaly Detect · Flags spikes and unusual event-volume patterns
modules/anomaly_detection.py
Output & State
Alerting, persistence, reporting, and presentation
- Alerting · Sends notable findings to logs, JSON, email, and desktop notifications
core/alerting.py - SQLite Store · Durable storage for logs, alerts, correlations, and statistics
core/database.py - Report · Generates summarized output artifacts from analyzed results
core/report.py - Dashboard · Rich-based interactive presentation of findings and trends
ui/dashboard.py
Inputs
Sample data and local reference assets
- Sample Log · Representative access-log input for passive analysis
data/sample_access.log - Blacklist · Local IP blacklist used by threat-intel checks
Connections 21
- main.py → Scanner: scan · Runs active host/network probing
- main.py → Log Analyzer: analyze · Runs passive log analysis
- main.py → Realtime Monitor: live mode · Enables continuous log monitoring
- Realtime Monitor → Log Ingestor: events · Pushes newly observed log content into ingestion
- Log Ingestor → Log Analyzer: entries · Supplies parsed log entries for detection
- Log Analyzer → Rule Loader: rules · Invokes custom detection rules
- Rule Loader → Example Rule: loads · Loads extension rule modules
- Scanner → Detection Engine: findings · Submits scan results for scoring
- Log Analyzer → Detection Engine: findings · Submits log-derived detections for scoring
- main.py → Detection Engine: correlate · Combines cross-source findings into higher-confidence incidents
- Detection Engine → Geo IP: enrich · Adds source-location context
- Detection Engine → Threat Intel: enrich · Adds IP reputation context
- Detection Engine → MITRE Map: map · Annotates detections with ATT&CK techniques
- Detection Engine → Anomaly Detect: score · Applies spike and anomaly context
- Threat Intel → Blacklist: local lookup · Uses local blacklist as a reputation source
- Detection Engine → Alerting: alerts · Sends notable incidents to notification channels
- Detection Engine → SQLite Store: persist · Stores alerts, correlations, and statistics
- Alerting → Report: output · Feeds generated artifacts and summaries
- SQLite Store → Dashboard: read · Provides stored results for presentation
- main.py → Report: summarize · Produces final run reports
- Sample Log → Log Ingestor: input · Representative log source for analysis