How phil155/lorawan_ids works
Project Type and Purpose
• This is a network‐based Intrusion Detection System (IDS) for LoRaWAN IoT devices.
• It integrates Suricata (packet‐level IDS) with a Python KNN anomaly detection pipeline, storing results in CrateDB and offering a simple GUI.
Main Architectural Layers & Components
A. External / Physical Layer
• LoRaWAN IoT devices → network traffic on interface (ens33)
B. Packet Capture & Detection (Suricata Engine)
• suricata.yaml (etc/suricata/) – engine configuration
• local.rules (etc/suricata/rules/) – IDS rules
• Suricata CLI (sudo suricata -c suricata.yaml -i ens33)
C. Event Export (Lua Plugin)
• local.lua (etc/suricata/lua-output/) – captures alerts, base64‐encodes payloads
• lua/bs64.lua, lua/lunajson (decoder, encoder, sax) – JSON serialization
D. Data Ingestion & Preprocessing (Python Layer)
• gateway.py / insert_packets.py – read JSON from Lua, push to pipeline
• dataset.py, 1_init.py – feature extraction, initial dataset creation
• 2_auto_intrusions.py, set_intrusion_auto.py – inject synthetic intrusions for labeling
• dataset_json/ – reference logs used for training/testing
E. Model Training & Prediction
• machine.py, 3_train_model.py – KNN model training
• predict_model.py – inference on new data streams
F. Storage & Query (CrateDB)
• sqlclient.py, insert_dataset.py – write features/predictions to CrateDB
• sql_help.sql – DDL/queries for schema setup
G. Presentation Layer
• GUI.py – simple interface to view alerts and anomaly scores
• result.txt / result_pred.txt – example output logs
Data Flow & Interactions
1) LoRaWAN devices → network interface ens33
2) Suricata captures packets → applies rules → triggers alerts
3) local.lua serializes alerts to JSON → writes to configured dir
4) Python gateway reads JSON → passes raw events to ingestion scripts
5) dataset.py & helpers extract features → labeled dataset in memory or on disk
6) train_model script builds KNN model → persists model object
7) predict_model script scores new events → anomaly vs. normal
8) sqlclient inserts feature records & prediction results into CrateDB
9) GUI fetches data from CrateDB → displays in tables/charts
Architectural Patterns & Principles
• Modular “pipe‐and‐filter” / ETL pipeline: each stage transforms data
• Clear separation of concerns:
• IDS engine (C/Lua), data formatting (Lua), ML pipeline (Python), storage (DB), visualization (GUI).
• Config‐driven deployment: YAML for Suricata, SQL for DB schema, constants.py for paths.
Technologies & Frameworks to Annotate
• Suricata (C-based IDS)
• Lua (scripting for JSON output) + lunajson, bs64 libraries
• Python3 ecosystem: standard library, scikit-learn (KNN) or custom KNN, sqlite or CrateDB client
• CrateDB (distributed SQL on JSON documents)
• Ubuntu 18.04 LTS (host OS)