muminur/grafana-loki-netwatch-monitoring-notification-by-mumin

1. Purpose and deployment model

NetWatch is a Python 3.11+ FastAPI application that sits inside a NOC workflow, not as a generic web app. Its job is to ingest network syslog events, turn them into actionable incidents, notify operators, and present the live state in a dashboard. It is designed to run either directly on a workstation/server or in Docker Compose, with SQLite as the durable store.

2. Primary event ingestion path

The main data flow is: Syslog → Grafana Loki → Grafana datasource proxy → NetWatch. NetWatch does not connect to Loki directly; it reads via Grafana’s proxy and can also receive UDP syslog. The receiver layer in src/core/syslog_receiver.py is responsible for fetching or consuming log lines and handing them into the processing pipeline.

3. Log processing pipeline

The core pipeline is split into clear stages:

- parser.py normalizes several IOS-XR syslog formats

- classifier.py maps parsed events to severity/mnemonic rules

- enricher.py adds device, interface, client, and AS context from static lookup data and cached external lookups

- correlator.py groups related events into incidents

- dedup.py suppresses repeated noise within time windows

This pipeline is the central architecture boundary: raw syslog becomes enriched, correlated incident records.

4. Correlation and incident model

Correlation is topology-aware rather than purely message-based. The system uses src/data/topology.py plus bundle/BGP mapping data to distinguish a true backhaul failure from unrelated alerts on the same device. The result is an incident layer that can contain root causes and symptoms, supports multiple concurrent incidents per device, and auto-resolves when recovery events arrive.

5. Persistence boundary: SQLite

src/database/ is the durable state boundary. SQLite in WAL mode stores alerts, incidents, acknowledgements, notification tracking, maintenance windows, settings, and statistics aggregates. The database is also the source of truth for restart recovery: active incidents, escalation state, shift handoff notes, and user acknowledgements are reconstructed from persisted rows.

6. Notification subsystem

src/notifications/ fans out significant events to external channels. Discord and Telegram are separate delivery adapters with shared formatting and escalation logic. Notifications are not just fire-and-forget: delivery status, retries, escalation after 15 minutes, and resolution messages are all tracked as part of the incident lifecycle.

7. API surface

src/api/routes.py exposes the operational REST endpoints: alerts, incidents, shift handoffs, statistics, topology, devices, settings, maintenance, and CSV export. src/api/websocket.py provides live browser push. The API is the boundary between the backend event engine and the operator-facing UI, and it also carries auth/rate-limiting rules when strict mode is enabled.

8. Web dashboard frontend

src/web/ is server-rendered Jinja2 pages plus static JavaScript/CSS. The frontend is not a separate SPA; it consumes the FastAPI endpoints and WebSocket stream. The JS modules handle chart rendering, topology interaction, sound alerts, shortcuts, and reconnecting live updates. The UI is purpose-built for NOC wall display, with live incident cards, severity tabs, and topology visualization.

9. Statistics and reporting

src/statistics/ is a separate analytical subsystem from the live alert stream. It computes hourly aggregation, health scoring, SLA-style metrics, and prefix exhaustion predictions. These calculations read from the persisted alert/incident history and are exposed through the API for the Statistics page and CSV export workflows.

10. Static reference data and enrichment tables

src/data/ contains the operational mapping layer that makes the system useful: device maps, interface descriptions, AS number references, topology, classification rules, and BGP bundle mappings. This is a significant architectural boundary because much of NetWatch’s value comes from converting generic log text into site- and link-specific operational context.

11. Operational concerns and safeguards

The repo includes rate limiting, optional API key auth, health and metrics endpoints, maintenance suppression, dedup windows, and self-monitoring of data freshness. These are not peripheral features; they are part of the runtime control plane that keeps the alerting pipeline reliable under load and during partial outages.

12. Infrastructure and release flow

The project is packaged for Docker with a persistent data volume and hardened runtime settings, and it also has install/restart scripts for Windows, macOS, and Linux. GitHub Actions runs CI across OS/Python combinations, and a deploy workflow pushes main to an Ubuntu server over SSH, rebuilds the container, and verifies /health. This makes deployment an explicit part of the architecture rather than an afterthought.

13. How to draw the system

A good architecture diagram should show four main zones:

- External sources: routers/syslog, Grafana/Loki, Discord, Telegram

- NetWatch backend: ingestion pipeline, correlation/dedup, notifications, statistics, API/WebSocket

- Persistence: SQLite and cached reference data

- Operator surface: dashboard, settings, incident handling, shift handoff

The most important arrows are the event pipeline into the backend, the write path into SQLite, and the live push path back to the browser and notification channels.