# muminur/grafana-loki-netwatch-monitoring-notification-by-mumin architecture

> Architecture diagram of the GitHub repository muminur/grafana-loki-netwatch-monitoring-notification-by-mumin, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin
- Repository: https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin

## Overview

1. **Purpose and deployment model**
   NetWatch is a Python 3.11+ FastAPI application that sits inside a NOC workflow, not as a generic web app. Its job is to ingest network syslog events, turn them into actionable incidents, notify operators, and present the live state in a dashboard. It is designed to run either directly on a workstation/server or in Docker Compose, with SQLite as the durable store.

2. **Primary event ingestion path**
   The main data flow is: **Syslog → Grafana Loki → Grafana datasource proxy → NetWatch**. NetWatch does not connect to Loki directly; it reads via Grafana’s proxy and can also receive UDP syslog. The receiver layer in `src/core/syslog_receiver.py` is responsible for fetching or consuming log lines and handing them into the processing pipeline.

3. **Log processing pipeline**
   The core pipeline is split into clear stages:
   - `parser.py` normalizes several IOS-XR syslog formats
   - `classifier.py` maps parsed events to severity/mnemonic rules
   - `enricher.py` adds device, interface, client, and AS context from static lookup data and cached external lookups
   - `correlator.py` groups related events into incidents
   - `dedup.py` suppresses repeated noise within time windows  
   This pipeline is the central architecture boundary: raw syslog becomes enriched, correlated incident records.

4. **Correlation and incident model**
   Correlation is topology-aware rather than purely message-based. The system uses `src/data/topology.py` plus bundle/BGP mapping data to distinguish a true backhaul failure from unrelated alerts on the same device. The result is an incident layer that can contain root causes and symptoms, supports multiple concurrent incidents per device, and auto-resolves when recovery events arrive.

5. **Persistence boundary: SQLite**
   `src/database/` is the durable state boundary. SQLite in WAL mode stores alerts, incidents, acknowledgements, notification tracking, maintenance windows, settings, and statistics aggregates. The database is also the source of truth for restart recovery: active incidents, escalation state, shift handoff notes, and user acknowledgements are reconstructed from persisted rows.

6. **Notification subsystem**
   `src/notifications/` fans out significant events to external channels. Discord and Telegram are separate delivery adapters with shared formatting and escalation logic. Notifications are not just fire-and-forget: delivery status, retries, escalation after 15 minutes, and resolution messages are all tracked as part of the incident lifecycle.

7. **API surface**
   `src/api/routes.py` exposes the operational REST endpoints: alerts, incidents, shift handoffs, statistics, topology, devices, settings, maintenance, and CSV export. `src/api/websocket.py` provides live browser push. The API is the boundary between the backend event engine and the operator-facing UI, and it also carries auth/rate-limiting rules when strict mode is enabled.

8. **Web dashboard frontend**
   `src/web/` is server-rendered Jinja2 pages plus static JavaScript/CSS. The frontend is not a separate SPA; it consumes the FastAPI endpoints and WebSocket stream. The JS modules handle chart rendering, topology interaction, sound alerts, shortcuts, and reconnecting live updates. The UI is purpose-built for NOC wall display, with live incident cards, severity tabs, and topology visualization.

9. **Statistics and reporting**
   `src/statistics/` is a separate analytical subsystem from the live alert stream. It computes hourly aggregation, health scoring, SLA-style metrics, and prefix exhaustion predictions. These calculations read from the persisted alert/incident history and are exposed through the API for the Statistics page and CSV export workflows.

10. **Static reference data and enrichment tables**
   `src/data/` contains the operational mapping layer that makes the system useful: device maps, interface descriptions, AS number references, topology, classification rules, and BGP bundle mappings. This is a significant architectural boundary because much of NetWatch’s value comes from converting generic log text into site- and link-specific operational context.

11. **Operational concerns and safeguards**
   The repo includes rate limiting, optional API key auth, health and metrics endpoints, maintenance suppression, dedup windows, and self-monitoring of data freshness. These are not peripheral features; they are part of the runtime control plane that keeps the alerting pipeline reliable under load and during partial outages.

12. **Infrastructure and release flow**
   The project is packaged for Docker with a persistent data volume and hardened runtime settings, and it also has install/restart scripts for Windows, macOS, and Linux. GitHub Actions runs CI across OS/Python combinations, and a deploy workflow pushes `main` to an Ubuntu server over SSH, rebuilds the container, and verifies `/health`. This makes deployment an explicit part of the architecture rather than an afterthought.

13. **How to draw the system**
   A good architecture diagram should show four main zones:
   - **External sources**: routers/syslog, Grafana/Loki, Discord, Telegram
   - **NetWatch backend**: ingestion pipeline, correlation/dedup, notifications, statistics, API/WebSocket
   - **Persistence**: SQLite and cached reference data
   - **Operator surface**: dashboard, settings, incident handling, shift handoff  
   The most important arrows are the event pipeline into the backend, the write path into SQLite, and the live push path back to the browser and notification channels.

Main entry points: [`src/api/routes.py`](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/api/routes.py) (REST API), [`src/core/correlator.py`](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/core/correlator.py) (Correlator), [`src/main.py`](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/main.py) (App bootstrap), [`src/core/enricher.py`](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/core/enricher.py) (Enricher)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_external["External sources"]
  node_routers_syslog(("Routers<br/>syslog source"))
  node_grafana_loki[("Grafana/Loki<br/>log proxy")]
  node_discord(("Discord<br/>alert channel"))
  node_telegram(("Telegram<br/>alert channel"))
end

subgraph group_backend["NetWatch backend"]
  node_main_app["App bootstrap<br/>FastAPI entrypoint<br/>[main.py]"]
  node_receiver["Syslog receiver<br/>ingestion<br/>[syslog_receiver.py]"]
  node_parser["Parser<br/>normalization<br/>[parser.py]"]
  node_classifier["Classifier<br/>severity mapping<br/>[classifier.py]"]
  node_enricher["Enricher<br/>context enrichment<br/>[enricher.py]"]
  node_correlator["Correlator<br/>incident engine<br/>[correlator.py]"]
  node_dedup["Dedup<br/>noise suppression<br/>[dedup.py]"]
  node_notifications["Notifier<br/>delivery orchestration<br/>[escalation.py]"]
  node_stats_engine["Stats engine<br/>analytics<br/>[engine.py]"]
  node_api_routes["REST API<br/>operational api<br/>[routes.py]"]
  node_api_ws["WebSocket<br/>live push<br/>[websocket.py]"]
  node_auth["Auth<br/>access control<br/>[auth.py]"]
  node_rate_limit["Rate limit<br/>control plane<br/>[rate_limit.py]"]
end

subgraph group_persistence["Persistence"]
  node_db_models[("DB models<br/>sqlite schema<br/>[models.py]")]
  node_db_crud["DB CRUD<br/>persistence access<br/>[crud.py]"]
  node_as_cache["AS cache<br/>lookup cache<br/>[as_cache.py]"]
  node_topology["Topology data<br/>reference data<br/>[topology.py]"]
  node_device_maps["Device maps<br/>reference data<br/>[device_map.py]"]
end

subgraph group_operator["Operator surface"]
  node_dashboard_page["Dashboard UI<br/>server-rendered page<br/>[dashboard.html]"]
  node_dashboard_js["Dashboard JS<br/>client interaction<br/>[dashboard.js]"]
  node_ws_client["Live client<br/>browser websocket<br/>[websocket.js]"]
end

subgraph group_release["Delivery"]
  node_deploy["Docker/CI<br/>delivery pipeline"]
end

node_routers_syslog -->|"syslog"| node_receiver
node_grafana_loki -->|"proxy log stream"| node_receiver
node_receiver -->|"normalize"| node_parser
node_parser -->|"parsed event"| node_classifier
node_classifier -->|"classified event"| node_enricher
node_enricher -->|"enriched event"| node_correlator
node_correlator -->|"incident candidates"| node_dedup
node_dedup -->|"persist"| node_db_crud
node_correlator -->|"incident lifecycle"| node_notifications
node_notifications -->|"deliver"| node_discord
node_notifications -->|"deliver"| node_telegram
node_enricher -->|"lookup"| node_device_maps
node_enricher -->|"cache lookup"| node_as_cache
node_correlator -->|"topology-aware"| node_topology
node_correlator -->|"incident state"| node_db_crud
node_stats_engine -->|"read history"| node_db_crud
node_api_routes -->|"read/write"| node_db_crud
node_api_routes -->|"metrics"| node_stats_engine
node_api_ws -->|"push"| node_dashboard_js
node_dashboard_page -->|"requests"| node_api_routes
node_dashboard_js -->|"subscribe"| node_api_ws
node_auth -->|"protect"| node_api_routes
node_rate_limit -->|"throttle"| node_api_routes
node_main_app -->|"wire"| node_receiver
node_main_app -->|"mount"| node_api_routes
node_main_app -->|"mount"| node_api_ws
node_main_app -.->|"run"| node_deploy
node_deploy -.->|"integrate"| node_grafana_loki
node_deploy -.->|"serve"| node_dashboard_page

click node_main_app "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/main.py"
click node_receiver "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/core/syslog_receiver.py"
click node_parser "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/core/parser.py"
click node_classifier "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/core/classifier.py"
click node_enricher "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/core/enricher.py"
click node_correlator "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/core/correlator.py"
click node_dedup "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/core/dedup.py"
click node_notifications "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/notifications/escalation.py"
click node_stats_engine "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/statistics/engine.py"
click node_api_routes "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/api/routes.py"
click node_api_ws "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/api/websocket.py"
click node_auth "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/auth.py"
click node_rate_limit "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/rate_limit.py"
click node_db_models "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/database/models.py"
click node_db_crud "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/database/crud.py"
click node_as_cache "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/database/as_cache.py"
click node_topology "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/data/topology.py"
click node_device_maps "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/data/device_map.py"
click node_dashboard_page "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/web/templates/dashboard.html"
click node_dashboard_js "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/web/static/js/dashboard.js"
click node_ws_client "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/web/static/js/websocket.js"
click node_deploy "https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/tree/main/Dockerfile"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_routers_syslog,node_grafana_loki,node_discord,node_telegram toneBlue
class node_main_app,node_receiver,node_parser,node_classifier,node_enricher,node_correlator,node_dedup,node_notifications,node_stats_engine,node_api_routes,node_api_ws,node_auth,node_rate_limit toneAmber
class node_db_models,node_db_crud,node_as_cache,node_topology,node_device_maps toneMint
class node_dashboard_page,node_dashboard_js,node_ws_client toneRose
class node_deploy toneIndigo
```

## Components

### External sources

Upstream event sources and outbound notification channels.

- **Routers**: syslog source · Network devices emitting IOS-XR syslog events.
- **Grafana/Loki**: log proxy · NetWatch reads syslog through Grafana's datasource proxy and can also receive UDP syslog.
- **Discord**: alert channel · Outbound incident and escalation notifications.
- **Telegram**: alert channel · Outbound incident and escalation notifications.

### NetWatch backend

Ingestion, correlation, notification, statistics, and API runtime.

- [App bootstrap](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/main.py): FastAPI entrypoint · Application startup, lifespan, and wiring for the runtime. · `src/main.py`
- [Syslog receiver](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/core/syslog_receiver.py): ingestion · Consumes syslog lines or fetches them from Grafana and feeds the pipeline. · `src/core/syslog_receiver.py`
- [Parser](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/core/parser.py): normalization · Normalizes IOS-XR syslog formats into structured events. · `src/core/parser.py`
- [Classifier](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/core/classifier.py): severity mapping · Maps parsed events to severity and mnemonic rules. · `src/core/classifier.py`
- [Enricher](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/core/enricher.py): context enrichment · Adds device, interface, client, and AS context from reference data and caches. · `src/core/enricher.py`
- [Correlator](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/core/correlator.py): incident engine · Groups related events into topology-aware incidents and resolves them on recovery. · `src/core/correlator.py`
- [Dedup](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/core/dedup.py): noise suppression · Suppresses repeated alerts within configurable time windows. · `src/core/dedup.py`
- [Notifier](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/notifications/escalation.py): delivery orchestration · Formats incidents and drives escalation and channel fan-out. · `src/notifications/escalation.py`
- [Stats engine](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/statistics/engine.py): analytics · Builds hourly aggregates, health scores, SLA metrics, and predictions. · `src/statistics/engine.py`
- [REST API](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/api/routes.py): operational api · Alerts, incidents, handoff, statistics, topology, settings, maintenance, and export endpoints. · `src/api/routes.py`
- [WebSocket](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/api/websocket.py): live push · Pushes live incident state to the browser. · `src/api/websocket.py`
- [Auth](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/auth.py): access control · Optional API key auth and strict-mode protections. · `src/auth.py`
- [Rate limit](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/rate_limit.py): control plane · Request throttling and runtime safeguards. · `src/rate_limit.py`

### Persistence

SQLite state and cached reference data used across restarts.

- [DB models](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/database/models.py): sqlite schema · Persistent incidents, alerts, acknowledgements, maintenance, settings, and metrics state. · `src/database/models.py`
- [DB CRUD](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/database/crud.py): persistence access · Write/read boundary for operational state and restart recovery. · `src/database/crud.py`
- [AS cache](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/database/as_cache.py): lookup cache · Cached external AS lookups supporting enrichment and topology context. · `src/database/as_cache.py`
- [Topology data](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/data/topology.py): reference data · Device and link topology used to distinguish root causes from symptoms. · `src/data/topology.py`
- [Device maps](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/data/device_map.py): reference data · Device, interface, and classification reference tables. · `src/data/device_map.py`

### Operator surface

Browser dashboard and live operational UI.

- [Dashboard UI](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/web/templates/dashboard.html): server-rendered page · Jinja2 operator console for live incidents and operational actions. · `src/web/templates/dashboard.html`
- [Dashboard JS](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/web/static/js/dashboard.js): client interaction · Charts, topology interactions, shortcuts, sounds, and reconnecting live updates. · `src/web/static/js/dashboard.js`
- [Live client](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/blob/main/src/web/static/js/websocket.js): browser websocket · Browser-side WebSocket consumer for real-time incident updates. · `src/web/static/js/websocket.js`

### Delivery

Containerization, install scripts, and CI/CD flow.

- [Docker/CI](https://github.com/muminur/grafana-loki-netwatch-monitoring-notification-by-mumin/tree/main/Dockerfile): delivery pipeline · Container build, compose runtime, install scripts, and GitHub Actions deploy flow. · `Dockerfile`

## Connections

- Routers → Syslog receiver: syslog · Raw syslog events enter NetWatch from network devices.
- Grafana/Loki → Syslog receiver: proxy log stream · NetWatch reads log lines through Grafana's datasource proxy.
- Syslog receiver → Parser: normalize · Receiver hands log lines into the parsing stage.
- Parser → Classifier: parsed event · Structured events are classified by severity and mnemonic.
- Classifier → Enricher: classified event · Classification feeds enrichment and context lookup.
- Enricher → Correlator: enriched event · Contextualized events are correlated into incidents.
- Correlator → Dedup: incident candidates · Related events are deduplicated and noise-suppressed.
- Dedup → DB CRUD: persist · Operational state and incident history are written to SQLite.
- Correlator → Notifier: incident lifecycle · Significant incident transitions trigger notifications and escalation.
- Notifier → Discord: deliver · Escalation and resolution messages go to Discord.
- Notifier → Telegram: deliver · Escalation and resolution messages go to Telegram.
- Enricher → Device maps: lookup · Enrichment uses device and interface reference data.
- Enricher → AS cache: cache lookup · External AS context is resolved through cached lookups.
- Correlator → Topology data: topology-aware · Topology information separates root causes from downstream symptoms.
- Correlator → DB CRUD: incident state · Incident and recovery state is persisted for restart recovery.
- Stats engine → DB CRUD: read history · Analytics reads alert and incident history from SQLite.
- REST API → DB CRUD: read/write · REST endpoints expose and mutate operational state.
- REST API → Stats engine: metrics · Statistics and reporting endpoints use the analytics engine.
- WebSocket → Dashboard JS: push · WebSocket streams live updates to the browser client.
- Dashboard UI → REST API: requests · Operator actions flow through the REST API.
- Dashboard JS → WebSocket: subscribe · Client reconnects and subscribes to live incident updates.
- Auth → REST API: protect · Optional auth guards the operational API.
- Rate limit → REST API: throttle · Request limiting protects the control plane.
- App bootstrap → Syslog receiver: wire · Application startup connects the ingestion pipeline.
- App bootstrap → REST API: mount · FastAPI registers operational routes.
- App bootstrap → WebSocket: mount · FastAPI registers live WebSocket delivery.
- App bootstrap → Docker/CI: run · The app is packaged and launched in container or installed runtime.
- Docker/CI → Grafana/Loki: integrate · Deployment assumes Grafana/Loki as the upstream log source.
- Docker/CI → Dashboard UI: serve · Deployment serves the operator dashboard and static assets.