# phil155/lorawan_ids architecture

> Architecture diagram of the GitHub repository phil155/lorawan_ids, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/phil155/lorawan_ids
- Repository: https://github.com/phil155/lorawan_ids

## Overview

##### Project Type and Purpose
- This is a network‐based Intrusion Detection System (IDS) for LoRaWAN IoT devices.
- It integrates Suricata (packet‐level IDS) with a Python KNN anomaly detection pipeline, storing results in CrateDB and offering a simple GUI.

##### Main Architectural Layers & Components
A. External / Physical Layer
- LoRaWAN IoT devices → network traffic on interface (ens33)
B. Packet Capture & Detection (Suricata Engine)
- suricata.yaml (etc/suricata/) – engine configuration
- local.rules (etc/suricata/rules/) – IDS rules
- Suricata CLI (`sudo suricata -c suricata.yaml -i ens33`)
C. Event Export (Lua Plugin)
- local.lua (etc/suricata/lua-output/) – captures alerts, base64‐encodes payloads
- lua/bs64.lua, lua/lunajson (decoder, encoder, sax) – JSON serialization
D. Data Ingestion & Preprocessing (Python Layer)
- gateway.py / insert_packets.py – read JSON from Lua, push to pipeline
- dataset.py, 1_init.py – feature extraction, initial dataset creation
- 2_auto_intrusions.py, set_intrusion_auto.py – inject synthetic intrusions for labeling
- dataset_json/ – reference logs used for training/testing
E. Model Training & Prediction
- machine.py, 3_train_model.py – KNN model training
- predict_model.py – inference on new data streams
F. Storage & Query (CrateDB)
- sqlclient.py, insert_dataset.py – write features/predictions to CrateDB
- sql_help.sql – DDL/queries for schema setup
G. Presentation Layer
- GUI.py – simple interface to view alerts and anomaly scores
- result.txt / result_pred.txt – example output logs

##### Data Flow & Interactions
1) LoRaWAN devices → network interface ens33
2) Suricata captures packets → applies rules → triggers alerts
3) local.lua serializes alerts to JSON → writes to configured `dir`
4) Python gateway reads JSON → passes raw events to ingestion scripts
5) dataset.py & helpers extract features → labeled dataset in memory or on disk
6) train_model script builds KNN model → persists model object
7) predict_model script scores new events → anomaly vs. normal
8) sqlclient inserts feature records & prediction results into CrateDB
9) GUI fetches data from CrateDB → displays in tables/charts

##### Architectural Patterns & Principles
- Modular “pipe‐and‐filter” / ETL pipeline: each stage transforms data
- Clear separation of concerns:
- IDS engine (C/Lua), data formatting (Lua), ML pipeline (Python), storage (DB), visualization (GUI).
- Config‐driven deployment: YAML for Suricata, SQL for DB schema, constants.py for paths.

##### Technologies & Frameworks to Annotate
- Suricata (C-based IDS)
- Lua (scripting for JSON output) + lunajson, bs64 libraries
- Python3 ecosystem: standard library, scikit-learn (KNN) or custom KNN, sqlite or CrateDB client
- CrateDB (distributed SQL on JSON documents)
- Ubuntu 18.04 LTS (host OS)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Layer
    subgraph "External Layer"
        Cloud["LoRaWAN IoT Devices"]:::network
    end

    %% Packet Capture & Detection
    subgraph "Packet Capture & Detection"
        SuricataEngine["Suricata Engine"]:::network
        ConfigYAML["suricata.yaml"]:::config
        Rules["local.rules"]:::config
        SuricataEngine --> ConfigYAML
        SuricataEngine --> Rules
    end

    Cloud -->|PCAP| SuricataEngine
    SuricataEngine -->|Alerts| LuaPlugin

    %% Event Export (Lua)
    subgraph "Event Export (Lua)"
        LuaPlugin["local.lua (Lua Plugin)"]:::processing
        BS64["bs64.lua"]:::processing
        LunajsonCore["lunajson.lua"]:::processing
        Decoder["decoder.lua"]:::processing
        Encoder["encoder.lua"]:::processing
        Sax["sax.lua"]:::processing
        LuaPlugin --> BS64
        LuaPlugin --> LunajsonCore
        LunajsonCore --> Decoder
        LunajsonCore --> Encoder
        LunajsonCore --> Sax
    end

    LuaPlugin -->|JSON event| Gateway

    %% Python Layer
    subgraph "Python Layer"
        subgraph "Configuration" 
            Constants["constants.py"]:::config
        end
        subgraph "Ingestion & Preprocessing" 
            Gateway["gateway.py"]:::processing
            InsertPackets["insert_packets.py"]:::processing
            DatasetInit["1_init.py"]:::processing
            AutoIntrusions["2_auto_intrusions.py"]:::processing
            SetIntrusion["set_intrusion_auto.py"]:::processing
            Dataset["dataset.py"]:::processing
            Constants --> Gateway
            Constants --> InsertPackets
            Gateway --> InsertPackets
            InsertPackets --> DatasetInit
            DatasetInit --> Dataset
            Dataset --> AutoIntrusions
            AutoIntrusions --> SetIntrusion
        end
        subgraph "Model Training & Prediction"
            Train["3_train_model.py"]:::processing
            Machine["machine.py"]:::processing
            Predict["predict_model.py"]:::processing
            Dataset --> Train
            Train --> Machine
            Machine --> Predict
        end
        subgraph "DB Client"
            SQLClient["sqlclient.py"]:::processing
            InsertDS["insert_dataset.py"]:::processing
            SQLHelp["sql_help.sql"]:::config
            Predict --> SQLClient
            SQLClient --> InsertDS
            SQLClient --> SQLHelp
        end
    end

    Gateway -->|Features| Dataset
    Predict -->|Predictions| SQLClient

    %% Storage & Query
    SQLClient -->|writes| DB["CrateDB"]:::storage

    %% Presentation Layer
    subgraph "Presentation Layer"
        GUI["GUI.py"]:::ui
        Logs1["result.txt"]:::ui
        Logs2["result_pred.txt"]:::ui
        GUI --> Logs1
        GUI --> Logs2
        GUI --> DB
    end

    %% Click Events
    click ConfigYAML "https://github.com/phil155/lorawan_ids/blob/main/etc/suricata/suricata.yaml"
    click Rules "https://github.com/phil155/lorawan_ids/blob/main/etc/suricata/rules/local.rules"
    click LuaPlugin "https://github.com/phil155/lorawan_ids/blob/main/etc/suricata/lua-output/local.lua"
    click BS64 "https://github.com/phil155/lorawan_ids/blob/main/lua/bs64.lua"
    click LunajsonCore "https://github.com/phil155/lorawan_ids/blob/main/lua/lunajson.lua"
    click Decoder "https://github.com/phil155/lorawan_ids/blob/main/lua/lunajson/decoder.lua"
    click Encoder "https://github.com/phil155/lorawan_ids/blob/main/lua/lunajson/encoder.lua"
    click Sax "https://github.com/phil155/lorawan_ids/blob/main/lua/lunajson/sax.lua"
    click Constants "https://github.com/phil155/lorawan_ids/blob/main/py/constants.py"
    click Gateway "https://github.com/phil155/lorawan_ids/blob/main/py/gateway.py"
    click InsertPackets "https://github.com/phil155/lorawan_ids/blob/main/py/insert_packets.py"
    click DatasetInit "https://github.com/phil155/lorawan_ids/blob/main/py/1_init.py"
    click AutoIntrusions "https://github.com/phil155/lorawan_ids/blob/main/py/2_auto_intrusions.py"
    click SetIntrusion "https://github.com/phil155/lorawan_ids/blob/main/py/set_intrusion_auto.py"
    click Dataset "https://github.com/phil155/lorawan_ids/blob/main/py/dataset.py"
    click Train "https://github.com/phil155/lorawan_ids/blob/main/py/3_train_model.py"
    click Machine "https://github.com/phil155/lorawan_ids/blob/main/py/machine.py"
    click Predict "https://github.com/phil155/lorawan_ids/blob/main/py/predict_model.py"
    click SQLClient "https://github.com/phil155/lorawan_ids/blob/main/py/sqlclient.py"
    click InsertDS "https://github.com/phil155/lorawan_ids/blob/main/py/insert_dataset.py"
    click SQLHelp "https://github.com/phil155/lorawan_ids/blob/main/py/sql_help.sql"
    click GUI "https://github.com/phil155/lorawan_ids/blob/main/py/GUI.py"
    click Logs1 "https://github.com/phil155/lorawan_ids/blob/main/py/result.txt"
    click Logs2 "https://github.com/phil155/lorawan_ids/blob/main/py/result_pred.txt"

    %% Styles
    classDef network fill:#e0ffe0,stroke:#009900;
    classDef processing fill:#e0e0ff,stroke:#0000cc;
    classDef storage fill:#ffe0b3,stroke:#cc6600;
    classDef ui fill:#f0e0ff,stroke:#800080;
    classDef config fill:#f9f9f9,stroke:#666666,font-style:italic;
```
