# r-v-abhishek/network-intrusion-detection-nosql architecture

> Architecture diagram of the GitHub repository r-v-abhishek/network-intrusion-detection-nosql, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/r-v-abhishek/network-intrusion-detection-nosql
- Repository: https://github.com/r-v-abhishek/network-intrusion-detection-nosql

## Overview

##### What this project is (type + purpose)
This is a **containerized, real-time Network Intrusion Detection System (NIDS)** implemented in **Python**, using:
- **Redis** for **real-time event streaming/caching**
- **Apache Cassandra** for **persistent storage** (network events + alerts)
- **Docker / Docker Compose** for deployment orchestration
- A lightweight **Dashboard API layer** (the `dashboard/` package) to expose alerts/analytics

##### Top-level “system” building blocks
- **`docker-compose.yml` / `Dockerfile`**: defines deployable services/containers (at minimum: Python app container, Redis, Cassandra).
- **`src/main.py`**: the primary application entrypoint—likely starts the pipeline and/or API server.
- **`streaming/`**: the core streaming/inference runtime:
- `pipeline_runner.py` — orchestrates the end-to-end pipeline
- `binary_inference.py` — binary classification (intrusion vs benign)
- `multiclass_inference.py` — attack-type classification
- **`dashboard/`**: API surfaces for consumers:
- `api_alerts.py` — alert querying/serving
- `api_analytics.py` — aggregate analytics endpoints
- **`config/`**: centralized configuration (`config.py`)—important because it defines how services connect (Redis/Cassandra hosts, ports, keyspaces, stream names, etc.).
- **`data/UNSW-NB15/sample.csv`, `unsw_feature_names.json`, `unsw_model_training.ipynb`**: offline dataset/training artifacts; these should appear as “offline training” or “reference/training” components (separate from realtime path).

##### Tests indicate intended seams
- `tests/test_pipeline.py` implies the pipeline is a distinct unit (good component boundary).
- `tests/test_dashboard_api.py` implies dashboard APIs are separable components.
- `tests/test_config.py` implies config is a foundational dependency across components.

##### Component / Module (Python packages and runtime flow inside the app container)

Optionally add:

This repo strongly benefits from separating **runtime architecture** (Redis/Cassandra/streaming) from **delivery architecture** (Jenkins/CI compose).

##### Important to capture explicitly
- Redis is **not just a cache** here; per README it’s **real-time event streaming**. Show it as a **message broker / stream** with directional flow.
- Cassandra is **system of record**. Show it as durable persistence and the source for analytics queries.

- `src/main.py` (Entrypoint / Service bootstrap)
- `config/config.py` (loads settings)
- `streaming/pipeline_runner.py` (starts pipeline loop)
- dashboard API modules (register routes / run server) if applicable

- `streaming/pipeline_runner.py`

- `dashboard/api_alerts.py`, `dashboard/api_analytics.py`

- **nids-app** (Python, built by `Dockerfile`)
- **redis**
- **cassandra**

- Network links between containers (same Docker network)
- Environment/config injection (env vars / config file) into nids-app
- Exposed ports:
- Dashboard API port (if it runs a web server)
- Redis/Cassandra internal ports (usually not exposed publicly in prod—show as internal unless the compose exposes them)

If compose defines volumes (common for Cassandra), show **Cassandra data volume** for persistence.

Given this repo, it’s better to split more than less. At minimum split into:
- Pipeline runner
- Binary inference
- Multiclass inference
- Redis (stream)
- Cassandra (persistence)
- Dashboard Alerts API
- Dashboard Analytics API
- Config module
- Offline training assets
- CI pipeline components

1. **System Context**: Users/SOC → Dashboard API; Telemetry Source → Detection System; external dependencies Redis/Cassandra.

## Diagram (Mermaid)

```mermaid
flowchart TD

%% =========================
%% Diagram 1 — System Context
%% =========================
subgraph "System Context (C4 L1)"
  direction TB

  SOC["Security Analyst / SOC User"]:::external
  Client["Dashboard Client (Browser / SOC Tool / curl)"]:::external
  Telemetry["Network Telemetry Source\n(packets/flows/features or CSV replay)"]:::external

  subgraph "NIDS Platform"
    direction TB
    NIDS["Network Intrusion Detection System (NIDS)"]:::system
    DashAPI["Dashboard API (Alerts + Analytics)"]:::api
    Stream["Redis Streams / PubSub\n(real-time event streaming + cache)"]:::stream
    Store[("Cassandra\n(system of record)")]:::db
  end

  SOC -->|"uses"| Client
  Client -->|"HTTP/REST"| DashAPI
  Telemetry -->|"network_events/features"| NIDS

  NIDS -->|"publish/consume\nRedisStreams"| Stream
  NIDS -->|"persist events+alerts\nCassandraCQL"| Store
  DashAPI -->|"query alerts+analytics\nCassandraCQL"| Store
end

%% ===============================
%% Diagram 2 — Container/Deployment
%% ===============================
subgraph "Container / Deployment (Docker Compose)"
  direction TB

  subgraph "Docker Host"
    direction TB

    Compose["Docker Compose\n(runtime services)"]:::ops
    AppC["nids-app container\n(Python Detection Service + Dashboard API)"]:::service
    RedisC["redis container\n(Redis Streams)"]:::stream
    CassC["cassandra container"]:::service
    CassDB[("Cassandra Data Volume")]:::volume

    Compose -->|"orchestrates"| AppC
    Compose -->|"orchestrates"| RedisC
    Compose -->|"orchestrates"| CassC

    AppC -.->|"loads config/env"| Cfg["Runtime Configuration"]:::config

    AppC -->|"Redis protocol\nStreams/PubSub"| RedisC
    AppC -->|"CassandraCQL"| CassC
    CassC -->|"persists"| CassDB

    Client2["Dashboard Client"]:::external
    Client2 -->|"HTTP/REST\n(exposed port)"| AppC
  end
end

%% =================================
%% Diagram 3 — Component/Module (App)
%% =================================
subgraph "Component / Module (Python App Container)"
  direction TB

  subgraph "src/"
    direction TB
    Main["main.py\n(Service bootstrap / entrypoint)"]:::service
  end

  subgraph "config/"
    direction TB
    CfgInit["__init__.py"]:::config
    CfgPy["config.py\n(Redis/Cassandra/streams/etc.)"]:::config
    CfgInit -->|"exports"| CfgPy
  end

  subgraph "streaming/"
    direction TB
    Runner["pipeline_runner.py\n(Pipeline Orchestrator)"]:::pipeline
    BinInf["binary_inference.py\n(Binary ML inference)"]:::ml
    MultiInf["multiclass_inference.py\n(Multiclass ML inference)"]:::ml

    Runner -->|"calls"| BinInf
    BinInf -->|"if intrusion"| MultiInf
  end

  subgraph "dashboard/"
    direction TB
    AlertsAPI["api_alerts.py\n(Alerts endpoints)"]:::api
    AnalyticsAPI["api_analytics.py\n(Analytics endpoints)"]:::api
  end

  subgraph "External Services (from containers)"
    direction TB
    RedisSvc["Redis Streams"]:::stream
    CassSvc[("Cassandra")]:::db
  end

  Telemetry2["Telemetry Ingest / Replay Job"]:::external

  %% Bootstrap + control/config
  Main -.->|"loads"| CfgInit
  Main -->|"starts"| Runner
  Main -->|"serves"| AlertsAPI
  Main -->|"serves"| AnalyticsAPI

  %% Realtime dataflow
  Telemetry2 -->|"network_events/features"| Runner
  Runner -->|"consume/publish\nRedisStreams"| RedisSvc
  Runner -->|"write events+alerts\nCassandraCQL"| CassSvc

  %% Serving queries
  AlertsAPI -->|"query alerts\nCassandraCQL"| CassSvc
  AnalyticsAPI -->|"query aggregates\nCassandraCQL"| CassSvc
end

%% ==================================
%% Diagram 4 — CI/CD (Jenkins + tests)
%% ==================================
subgraph "CI/CD (Delivery Architecture)"
  direction TB

  Dev["Developer (git push/PR)"]:::external
  Jenkins["Jenkins"]:::cicd
  Pipe["Jenkins Pipeline\n(stages: test/build)"]:::cicd

  ComposeCI["docker-compose.ci.yml\n(CI integration services)"]:::ops
  RedisCI["CI Redis service"]:::stream
  CassCI["CI Cassandra service"]:::service

  Tests["tests/\n(unit+integration)"]:::test
  TPipe["test_pipeline.py"]:::test
  TDash["test_dashboard_api.py"]:::test
  TCfg["test_config.py"]:::test

  DockerfileNode["Dockerfile\n(build app image)"]:::ops
  ComposeNode["docker-compose.yml\n(runtime orchestration)"]:::ops
  Docs["CI/CD Docs"]:::doc
  CICDmd["CICD.md"]:::doc
  JSetup["JENKINS_SETUP.md"]:::doc
  JenkinsfileNode["Jenkinsfile"]:::cicd

  Dev -->|"triggers"| Jenkins
  Jenkins -->|"runs"| Pipe
  Jenkins -->|"defined by"| JenkinsfileNode

  Pipe -->|"spins up"| ComposeCI
  ComposeCI -->|"starts"| RedisCI
  ComposeCI -->|"starts"| CassCI

  Pipe -->|"executes"| Tests
  Tests -->|"includes"| TPipe
  Tests -->|"includes"| TDash
  Tests -->|"includes"| TCfg

  Pipe -->|"builds"| DockerfileNode
  Pipe -->|"packages deploy config"| ComposeNode

  Pipe -.->|"references"| CICDmd
  Pipe -.->|"references"| JSetup
  Docs -->|"contains"| CICDmd
  Docs -->|"contains"| JSetup

  %% CI tests use integration services
  Tests -->|"Redis protocol"| RedisCI
  Tests -->|"CassandraCQL"| CassCI
end

%% Offline artifacts (referenced in explanation)
subgraph "Offline Training / Reference (separate from realtime)"
  direction TB
  DataCSV["UNSW-NB15 sample.csv"]:::offline
  FeatSchema["Feature Schema\nunsw_feature_names.json"]:::offline
  TrainNB["Model Training Notebook\nunsw_model_training.ipynb"]:::offline
  ModelArt["Model Artifacts\n(trained weights/encoders)"]:::offline

  DataCSV -->|"training data"| TrainNB
  FeatSchema -->|"feature list"| TrainNB
  TrainNB -->|"produces"| ModelArt

  ModelArt -.->|"loaded by inference modules"| BinInf
  ModelArt -.->|"loaded by inference modules"| MultiInf
end

%% =========================
%% Click Events (file mapping)
%% =========================
click ComposeNode "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/docker-compose.yml"
click ComposeCI "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/docker-compose.ci.yml"
click DockerfileNode "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/tree/master/Dockerfile"

click Main "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/src/main.py"

click Runner "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/streaming/pipeline_runner.py"
click BinInf "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/streaming/binary_inference.py"
click MultiInf "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/streaming/multiclass_inference.py"

click AlertsAPI "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/dashboard/api_alerts.py"
click AnalyticsAPI "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/dashboard/api_analytics.py"

click CfgPy "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/config/config.py"
click CfgInit "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/config/__init__.py"

click DataCSV "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/data/UNSW-NB15/sample.csv"
click FeatSchema "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/unsw_feature_names.json"
click TrainNB "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/unsw_model_training.ipynb"

click JenkinsfileNode "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/tree/master/Jenkinsfile"
click CICDmd "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/CICD.md"
click JSetup "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/JENKINS_SETUP.md"

click TPipe "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/tests/test_pipeline.py"
click TDash "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/tests/test_dashboard_api.py"
click TCfg "https://github.com/r-v-abhishek/network-intrusion-detection-nosql/blob/master/tests/test_config.py"

%% =========================
%% Styles (color + shapes)
%% =========================
classDef external fill:#f5f5f5,stroke:#616161,color:#111,stroke-width:1px;
classDef system fill:#e3f2fd,stroke:#1565c0,color:#0d47a1,stroke-width:1.5px;
classDef service fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20,stroke-width:1.5px;
classDef pipeline fill:#fff8e1,stroke:#ff8f00,color:#5d4037,stroke-width:1.5px;
classDef ml fill:#f3e5f5,stroke:#6a1b9a,color:#4a148c,stroke-width:1.5px;
classDef api fill:#e0f7fa,stroke:#006064,color:#00363a,stroke-width:1.5px;
classDef stream fill:#fff3e0,stroke:#ef6c00,color:#e65100,stroke-width:1.5px;
classDef db fill:#ffebee,stroke:#c62828,color:#b71c1c,stroke-width:1.5px;
classDef config fill:#ede7f6,stroke:#4527a0,color:#311b92,stroke-width:1.5px;
classDef ops fill:#eceff1,stroke:#37474f,color:#263238,stroke-width:1.5px;
classDef volume fill:#fce4ec,stroke:#ad1457,color:#880e4f,stroke-width:1.5px;
classDef cicd fill:#e8eaf6,stroke:#283593,color:#1a237e,stroke-width:1.5px;
classDef test fill:#f1f8e9,stroke:#558b2f,color:#33691e,stroke-width:1.5px;
classDef doc fill:#fafafa,stroke:#424242,color:#212121,stroke-width:1.5px,stroke-dasharray: 4 3;
classDef offline fill:#fafafa,stroke:#757575,color:#212121,stroke-width:1.5px,stroke-dasharray: 6 4;
```
