How ductrongx/log-monitoring-system works
Log Monitoring System centralizes application logs for real-time processing, search, monitoring, and response. Applications submit logs through the ingestion API; ingestion validates access, sanitizes and publishes raw-log events, which workers normalize and store while emitting live, critical-alert, and anomaly signals. Detection and alerting lead to notifications and optional AI-assisted incident investigation; engineers use APIs and the dashboard for search, analytics, identity, and retention management. Paths for unsampled subsystems identify existing module directories; their internal wiring is shown only where documented.
Main entry points: alerting/internal Alert evaluation · processing/internal Processing worker · anomaly/internal Anomaly detection · auth-storage.ts Monitoring dashboard
Components 27
Ingestion and processing
- Log ingestion API
apps/backend/src/main/java/com/vdt/log_monitoring/api/ingestion/LogIngestionController.java - Ingestion service
apps/backend/src/main/java/com/vdt/log_monitoring/modules/ingestion/internal/LogIngestionService.java - Raw log queue
- Processing worker
apps/backend/src/main/java/com/vdt/log_monitoring/modules/processing/internal - ClickHouse logs
- Live log events
Detection and response
- Critical alert events
- Anomaly signals
- Anomaly detection
apps/backend/src/main/java/com/vdt/log_monitoring/modules/anomaly/internal - Anomaly reports
- Detected events
- Alert evaluation
apps/backend/src/main/java/com/vdt/log_monitoring/modules/alerting/internal - Alert state
- Incident investigation
apps/backend/src/main/java/com/vdt/log_monitoring/modules/incident/internal - Incident AI service
apps/backend/src/main/java/com/vdt/log_monitoring/modules/incident/internal/ai/IncidentAnomalyReportAiService.java
Query and operations
- WebSocket delivery
apps/backend/src/main/java/com/vdt/log_monitoring/api/realtime - Monitoring dashboard
apps/frontend/src/features/auth/auth-storage.ts - Search and analytics
apps/backend/src/main/java/com/vdt/log_monitoring/modules/analytics/internal - Retention policies
apps/backend/src/main/java/com/vdt/log_monitoring/modules/retention/internal/RetentionService.java
Identity and access
- Identity and access API
apps/backend/src/main/java/com/vdt/log_monitoring/api/identity - Identity services
apps/backend/src/main/java/com/vdt/log_monitoring/modules/identity/internal - Prometheus target API
apps/backend/src/main/java/com/vdt/log_monitoring/api/identity/PrometheusTargetController.java
Other components
- Application sources
- Prometheus metrics
- Redis state
- Telegram
- PostgreSQL
Connections 33
- Application sources → Log ingestion API: submit logs
- Log ingestion API → Ingestion service: invoke facade
- Ingestion service → Identity services: verify API key
- Ingestion service → Raw log queue: publish raw logs
- Raw log queue → Processing worker: consume logs
- Processing worker → ClickHouse logs: store processed logs
- Processing worker → Live log events: publish live events
- Processing worker → Critical alert events: publish critical events
- Processing worker → Anomaly signals: publish anomaly signals
- Anomaly signals → Anomaly detection: consume signals
- Anomaly detection → Prometheus metrics: query metrics
- Anomaly detection → Anomaly reports: store reports
- Anomaly detection → Detected events: publish detections
- Detected events → Alert evaluation: consume detections
- Critical alert events → Alert evaluation: consume alerts
- Alert evaluation → Alert state: store occurrences
- Alert evaluation → Redis state: deduplicate alerts
- Alert evaluation → Telegram: send notifications
- Alert evaluation → WebSocket delivery: publish alert events
- Live log events → WebSocket delivery: deliver live logs
- WebSocket delivery → Monitoring dashboard: push events
- Monitoring dashboard → Search and analytics: search logs
- Search and analytics → ClickHouse logs: query logs
- Monitoring dashboard → Identity and access API: management requests
- Identity and access API → Identity services: invoke services
- Identity services → PostgreSQL: persist identity
- Prometheus target API → Prometheus metrics: provide scrape targets
- Detected events → Incident investigation: trigger investigation
- Incident investigation → Incident AI service: request analysis
- Incident AI service → Incident investigation: return analysis
- Monitoring dashboard → Retention policies: manage retention
- Retention policies → PostgreSQL: read policies and runs
- Retention policies → ClickHouse logs: delete expired logs