How mus1afq/hybrid-ids-containers works
1. Overall purpose
This repository implements a late-fusion intrusion detection system for Docker containers. It treats syscall telemetry and network telemetry as two separate detection channels, then combines their probability outputs with a deterministic rule to produce a final benign/malicious verdict.
2. Main architecture boundary
The system is split into two largely independent modality pipelines:
• Syscall pipeline: captures or loads syscall-derived features, trains a Logistic Regression model, and outputs a malicious probability.
• Network pipeline: captures or loads network-flow features, trains a Random Forest model, and outputs a malicious probability.
A third hybrid layer consumes both probabilities and applies fixed fusion thresholds.
3. Training data sources
The repo uses two static datasets as the main training/evaluation sources:
• data/raw/syscall/ for syscall modeling, based on CHIDS
• data/raw/network/ for network modeling, based on Bot-IoT
These are separate domains, which is important for the architecture because each model learns from a different feature space and data distribution.
4. Model artifacts
Trained artifacts are stored under models/:
• models/syscall/best_model.pkl plus scaler.pkl
• models/network/best_network_model.pkl
The syscall pipeline depends on a preprocessing scaler, while the network pipeline appears to use a single serialized model artifact.
5. Runtime capture subsystem
The live-demo path is in src/hybrid/runtime/. It captures telemetry from active Docker containers using:
• sysdig for syscall collection
• tshark for packet capture
The shell script run_and_capture.sh orchestrates capture, and the runtime raw logs/pcaps land under data/runtime/logs/.
6. Runtime feature extraction and schema alignment
After capture, separate extractors convert raw telemetry into model-ready features:
• extract_syscall_features.py
• extract_network_features.py
Then alignment scripts normalize the runtime feature schema to match what the trained models expect:
• align_syscall_schema.py
• align_network_schema.py
This boundary is important: the repo distinguishes raw capture, feature extraction, and schema compatibility as separate steps.
7. Inference boundary
Inference is also split by modality:
• src/syscall/inference/run_syscall_inference.py
• src/network/inference/run_network_inference.py
Each script loads its model artifact and writes prediction CSVs under outputs/predictions/. The syscall pipeline also uses the saved scaler before inference.
8. Hybrid fusion layer
The fusion subsystem is in src/hybrid/fusion/run_hybrid_fusion.py. It joins syscall and network predictions by run identifier and applies a rule-based late-fusion decision using probability thresholds rather than a learned meta-model. This means the hybrid result is not a third ML model; it is a deterministic decision layer on top of the two base models.
9. Evaluation regimes
There are two distinct evaluation tracks:
• Synthetic / offline validation: compares single-modality and hybrid performance on prepared CSVs.
• Runtime evaluation: scores the end-to-end live Docker demo scenarios.
These are handled in src/*/evaluation/ and produce metrics in outputs/metrics/ and plots in outputs/plots/.
10. Outputs and reporting
The repo persists artifacts in three main output areas:
• outputs/predictions/ for per-model and hybrid prediction CSVs
• outputs/metrics/ for summary metrics tables and text reports
• outputs/plots/ for confusion matrices, ROC curves, and window-analysis figures
This makes the project reproducible and also supports dissertation-style reporting.
11. Demo scenarios
The demo/ directory contains several prepared runtime scenario scripts that likely drive different container behaviors for live testing. These scenarios feed the runtime capture pipeline and demonstrate how the system behaves on benign and malicious workloads.
12. Validation and test boundary
The tests/ directory contains end-to-end validation scripts for datasets, synthetic evaluations, and hybrid behavior. This indicates the repo is organized around pipeline correctness across stages, not just model accuracy.
13. Key technology choices that shape the architecture
The important technologies here are:
• Python 3.11+
• scikit-learn for the classifiers
• joblib/pickle-style serialization for model artifacts
• Docker as the runtime environment under observation
• sysdig and tshark as capture tools
These are not incidental; they define how data enters the system and how models are deployed.
14. Conceptual diagram guidance
If you are drawing the architecture, the clearest structure is:
Container workload → capture tools → raw telemetry → feature extraction/schema alignment → modality-specific models → probability outputs → fusion rule → final verdict, with separate branches for offline training/evaluation and live runtime capture.
Main entry points: run_hybrid_fusion.py Late fusion · run_syscall_inference.py Syscall infer · run_network_inference.py Network infer · tests Validation
Components 21
Data
Static datasets and runtime telemetry artifacts
- Syscall data · CHIDS-derived offline and synthetic syscall datasets
data/raw/syscall - Network data · Bot-IoT-derived offline and synthetic network datasets
data/raw/network - Runtime logs · Captured syscall logs, pcaps, and aligned runtime CSVs
data/runtime
Models
Serialized trained artifacts
- Syscall model · Saved syscall classifier plus scaler
models/syscall - Network model · Saved network classifier artifact
models/network
Syscall pipeline
Syscall-based detection branch
- Syscall train · Trains the syscall model and scaler
src/syscall/training/train_syscall_model.py - Syscall infer · Loads syscall artifact and emits prediction CSVs
src/syscall/inference/run_syscall_inference.py - Syscall eval · Offline and synthetic syscall scoring
src/syscall/evaluation/evaluate_syscall.py
Network pipeline
Network-flow detection branch
- Network train · Trains the network classifier
src/network/training/train_network_model.py - Network infer · Loads network artifact and emits prediction CSVs
src/network/inference/run_network_inference.py - Network eval · Offline and synthetic network scoring
src/network/evaluation/evaluate_network.py
Hybrid layer
Runtime capture, feature alignment, and late fusion
- Capture · Orchestrates sysdig and tshark collection from Docker workloads
src/hybrid/runtime/run_and_capture.sh - Feature align · Extracts telemetry features and aligns runtime schemas
src/hybrid/runtime - Late fusion · Combines modality probabilities with threshold rules
src/hybrid/fusion/run_hybrid_fusion.py - Hybrid eval · Synthetic and runtime hybrid assessment
src/hybrid/evaluation
Outputs
Predictions, metrics, and plots
- Predictions · Per-model and hybrid prediction CSV outputs
outputs/predictions - Metrics · Summary metrics and report tables
outputs/metrics - Plots · Confusion matrices, ROC curves, and analysis figures
outputs/plots
Validation
End-to-end checks and scenario drivers
- Demo scenarios · Prepared benign and malicious container workloads
demo - Validation · Dataset, synthetic, and end-to-end pipeline checks
tests
Other components
- Config · Repository-wide settings and path wiring
src/common/config.py
Connections 31
- Config → Syscall train: paths · Shared configuration used during training
- Config → Network train: paths · Shared configuration used during training
- Config → Capture: paths · Runtime capture and file locations
- Syscall data → Syscall train: train data · Offline syscall datasets feed model training
- Network data → Network train: train data · Offline network datasets feed model training
- Syscall train → Syscall model: artifact · Produces classifier and scaler artifacts
- Network train → Network model: artifact · Produces trained classifier artifact
- Syscall model → Syscall infer: load · Inference loads saved syscall model and scaler
- Network model → Network infer: load · Inference loads saved network model
- Syscall data → Syscall eval: validate · Offline and synthetic syscall evaluation inputs
- Network data → Network eval: validate · Offline and synthetic network evaluation inputs
- Syscall infer → Predictions: writes · Stores syscall prediction CSVs
- Network infer → Predictions: writes · Stores network prediction CSVs
- Capture → Runtime logs: raw telemetry · Captures logs and packet traces from Docker workloads
- Demo scenarios → Capture: drives · Scenario scripts trigger live container behaviors
- Runtime logs → Feature align: input · Raw runtime telemetry is converted into model-ready features
- Feature align → Syscall infer: aligned features · Aligned syscall runtime features feed inference
- Feature align → Network infer: aligned features · Aligned network runtime features feed inference
- Syscall infer → Late fusion: probability · Supplies syscall malicious probability
- Network infer → Late fusion: probability · Supplies network malicious probability
- Late fusion → Hybrid eval: final verdict · Late-fusion decision is evaluated
- Late fusion → Predictions: writes · Stores hybrid prediction CSVs
- Hybrid eval → Metrics: reports · Produces hybrid metrics tables
- Syscall eval → Metrics: reports · Produces syscall metrics tables
- Network eval → Metrics: reports · Produces network metrics tables
- Syscall eval → Plots: figures · Produces syscall analysis plots
- Network eval → Plots: figures · Produces network analysis plots
- Hybrid eval → Plots: figures · Produces hybrid analysis plots
- Validation → Syscall data: checks · Validates syscall datasets and synthetic cases
- Validation → Network data: checks · Validates network datasets and synthetic cases
- Validation → Late fusion: checks · Validates hybrid behavior and end-to-end flow