How anish-dev09/ics-networks works
1. This repository is organized as a full ICS intrusion-detection pipeline, not just a single model. The main flow is: HAI dataset ingestion from data/raw/ → preprocessing and feature/sequence generation in src/ → model training/evaluation scripts at the repo root → persisted artifacts in results/ → Streamlit-based demo UI.
2. The data boundary is clear. Raw HAI dataset files live under data/raw/hai, while derived CNN-ready arrays are stored in data/processed/cnn_sequences as NumPy tensors plus metadata JSON. That split indicates two parallel preparation paths: one for tabular ML models and one for temporal deep-learning models.
3. src/data is the ingestion layer. hai_loader.py loads and slices the HAI dataset, data_loader.py provides general loading utilities, and sequence_generator.py transforms sensor time series into sliding windows for the CNN pipeline. This is the handoff point between raw CSV data and model-ready inputs.
4. src/features is the engineered-feature subsystem for classical ML. feature_engineering.py and ics_feature_extractor.py build statistical, temporal, lag, and correlation-based features from sensor streams. The repo’s Random Forest and XGBoost models depend on this layer rather than on raw sequences.
5. src/models contains the detector implementations themselves. baseline_detector.py provides anomaly baselines such as Isolation Forest, Z-score, and IQR rules; ml_models.py wraps Random Forest and XGBoost; cnn_models.py defines the 1D-CNN architecture; deep_anomaly_detector.py, ensemble_detector.py, and encrypted_traffic_detector.py suggest additional higher-level detection variants; and protocol_validator.py likely enforces protocol-aware sanity checks before scoring.
6. Protocol awareness is a separate subsystem under src/protocols. Parsers for Modbus, DNP3, and S7comm, plus protocol_detector.py, show that the project can inspect industrial network traffic at the application/protocol layer rather than treating all flows as generic packets. This is an important architecture boundary because protocol parsing feeds specialized validation and encrypted-traffic handling.
7. The orchestration layer lives in src/detection. integrated_detector.py and advanced_integrated_detector.py are the likely top-level coordinators that combine protocol parsing, feature extraction, temporal context, model inference, and post-processing/thresholding into a single detection decision. This is where the repository moves from “individual models” to “system-level detector.”
8. src/utils holds the adaptive and operational logic around the detectors. Modules such as adaptive_threshold.py, temporal_analyzer.py, advanced_temporal_context.py, online_learning_system.py, automatic_pattern_learning.py, benign_pattern_learner.py, class_imbalance_handler.py, and performance_optimizer.py indicate support for concept drift, imbalance handling, temporal context, and runtime tuning. These utilities sit beside the models rather than inside them, which keeps adaptation concerns separate from core inference code.
9. Training and evaluation are driven by repository-level scripts. train_ml_models.py, train_cnn_model.py, prepare_cnn_data.py, compare_models.py, optimize_thresholds.py, train_zero_day_detector.py, and quick_test_baseline.py form the offline pipeline entry points. They are responsible for generating artifacts and metrics rather than serving live traffic.
10. Persisted outputs are split into model artifacts and evaluation artifacts under results/. results/models contains trained binaries and their companion metadata, including the CNN Keras model, ML pickles, and feature-engineering objects; results/metrics contains CSV summaries for model comparison, baseline results, and feature importance. That makes the repository reproducible: the training code produces both inference assets and auditability data.
11. The demo boundary is separate from training. streamlit_app.py and demo/app.py provide the UI layer, while demo/mock_data.py and demo/mock_hai_data.py supply synthetic/sample traffic for interactive exploration. This suggests the live dashboard is designed to visualize detector behavior without requiring the full training pipeline to run inside the app.
12. Configuration and environment management are lightweight but explicit. configs/config.yaml centralizes settings, src/utils/config_utils.py loads them, and .devcontainer plus .streamlit/config.toml show the project is set up for both containerized development and Streamlit deployment. The repository is therefore structured for offline research workflows and a separate presentation/demo runtime.
13. Verification is covered by focused tests in the repo root: test_feature_engineering.py, test_model_loading.py, and test_protocols.py. These correspond to the key architecture seams: feature generation, serialized model compatibility, and protocol parsing correctness.
14. In diagram terms, the most important boundaries are: raw HAI data vs. derived feature/sequence data, tabular ML vs. sequence-based CNN processing, protocol parsing vs. generic sensor modeling, and offline training/evaluation vs. interactive Streamlit demo. Those are the core subsystems that define how data moves through the repository.
Main entry points: integrated_detector.py Integrated detector · models Model artifacts · protocol_detector.py Protocol detector · hai_loader.py HAI loader
Components 31
Data pipeline
Raw HAI ingestion and CNN-ready sequence preparation.
- HAI raw data · Source HAI dataset files.
data/raw/hai - Data loader · General data loading utilities.
src/data/data_loader.py - HAI loader · HAI-specific loading and slicing.
src/data/hai_loader.py - Sequence gen · Sliding-window generation for CNN inputs.
src/data/sequence_generator.py - CNN sequences · Prepared NumPy tensors and metadata for temporal models.
data/processed/cnn_sequences
Feature engineering
Classical ML feature construction from sensor streams.
- Feature engineering · Statistical, temporal, lag, and correlation features.
src/features/feature_engineering.py - ICS extractor · ICS-specific feature extraction from sensor streams.
src/features/ics_feature_extractor.py
Detectors
Baseline, ML, CNN, and ensemble detector implementations.
- Baseline detector · Isolation Forest, Z-score, and IQR baselines.
src/models/baseline_detector.py - ML models · Random Forest and XGBoost detector wrappers.
src/models/ml_models.py - CNN models · 1D-CNN architecture for sequence-based detection.
src/models/cnn_models.py - Ensemble detector · Combines multiple detector outputs.
src/models/ensemble_detector.py - Deep anomaly · Higher-level anomaly detection variant.
src/models/deep_anomaly_detector.py
Protocol layer
Industrial protocol parsing and protocol-aware validation.
- Protocol detector · Identifies protocol-specific traffic handling.
src/protocols/protocol_detector.py - Modbus parser · Parses Modbus traffic.
src/protocols/modbus_parser.py - DNP3 parser · Parses DNP3 traffic.
src/protocols/dnp3_parser.py - S7comm parser · Parses S7comm traffic.
src/protocols/s7comm_parser.py
Orchestration
Integrated detection and runtime adaptation logic.
- Integrated detector · Combines parsing, features, models, and scoring.
src/detection/integrated_detector.py - Advanced detector · Expanded coordinated detection pipeline.
src/detection/advanced_integrated_detector.py - Temporal analyzer · Temporal context extraction and analysis.
src/utils/temporal_analyzer.py - Adaptive threshold · Dynamic threshold adjustment.
src/utils/adaptive_threshold.py - Online learning · Incremental adaptation and drift handling.
src/utils/online_learning_system.py
Training and eval
Offline scripts that build models and compare results.
- Train ML · Offline training entry point for classical models.
train_ml_models.py - Train CNN · Offline training entry point for CNN models.
train_cnn_model.py - Prepare CNN · Builds CNN-ready processed arrays.
prepare_cnn_data.py - Compare models · Aggregates model comparison metrics.
compare_models.py - Optimize thresholds · Searches decision thresholds.
optimize_thresholds.py - Zero-day train · Trains specialized zero-day detector flow.
train_zero_day_detector.py
Demo UI
Streamlit presentation layer and sample data.
- Streamlit app · Primary demo interface.
streamlit_app.py - Demo app · Secondary demo entry point.
demo/app.py
Persisted artifacts
Saved models, metadata, and evaluation outputs.
- Model artifacts · Persisted trained models and metadata.
results/models - Metrics store · Persisted evaluation summaries and comparisons.
results/metrics
Connections 31
- HAI raw data → Data loader: load · Raw HAI data is read by generic loaders.
- Data loader → HAI loader: specialize · HAI-specific slicing builds on the general loader.
- HAI loader → Feature engineering: tabular path · Loaded sensor data feeds engineered features.
- HAI loader → Sequence gen: sequence path · Loaded sensor data feeds sliding windows.
- Sequence gen → CNN sequences: persist · Generated arrays are stored for CNN training.
- Feature engineering → ICS extractor: extend · ICS-specific extraction enriches engineered features.
- ICS extractor → ML models: features · Classical ML models consume engineered features.
- Feature engineering → ML models: features · Feature matrices drive classical detectors.
- Sequence gen → CNN models: sequences · Temporal windows feed the CNN architecture.
- Baseline detector → Ensemble detector: baseline input · Baseline scores can be fused into ensembles.
- ML models → Ensemble detector: model input · Classical model outputs contribute to fusion.
- CNN models → Deep anomaly: deep input · CNN outputs can be wrapped by deeper anomaly logic.
- Protocol detector → Modbus parser: route · Protocol-aware routing can select Modbus parsing.
- Protocol detector → DNP3 parser: route · Protocol-aware routing can select DNP3 parsing.
- Protocol detector → S7comm parser: route · Protocol-aware routing can select S7comm parsing.
- Modbus parser → Integrated detector: parse · Parsed protocol context informs integrated detection.
- DNP3 parser → Integrated detector: parse · Parsed protocol context informs integrated detection.
- S7comm parser → Integrated detector: parse · Parsed protocol context informs integrated detection.
- Temporal analyzer → Integrated detector: context · Temporal context supports scoring decisions.
- Adaptive threshold → Integrated detector: thresholds · Adaptive thresholds tune integrated scoring.
- Online learning → Advanced detector: adapt · Online learning supports advanced orchestration.
- Integrated detector → Advanced detector: compose · Advanced detector builds on the integrated detector.
- Prepare CNN → CNN sequences: generate · Preparation script creates the processed CNN arrays.
- Train ML → Model artifacts: save · Classical training writes model artifacts.
- Train CNN → Model artifacts: save · CNN training writes model artifacts.
- Compare models → Metrics store: write · Model comparison outputs metrics summaries.
- Optimize thresholds → Metrics store: write · Threshold search writes evaluation outputs.
- Zero-day train → Model artifacts: save · Zero-day training produces persisted detectors.
- Model artifacts → Streamlit app: load · The demo consumes saved models for visualization.
- Model artifacts → Demo app: load · The demo consumes saved models for visualization.
- Streamlit app → Demo app: share UI · Both entry points represent the presentation layer.