# mus1afq/hybrid-ids-containers architecture

> Architecture diagram of the GitHub repository mus1afq/hybrid-ids-containers, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mus1afq/hybrid-ids-containers
- Repository: https://github.com/mus1afq/hybrid-ids-containers

## Overview

1. **Overall purpose**
This repository implements a **late-fusion intrusion detection system for Docker containers**. It treats **syscall telemetry** and **network telemetry** as two separate detection channels, then combines their probability outputs with a deterministic rule to produce a final benign/malicious verdict.

2. **Main architecture boundary**
The system is split into two largely independent modality pipelines:
- **Syscall pipeline**: captures or loads syscall-derived features, trains a **Logistic Regression** model, and outputs a malicious probability.
- **Network pipeline**: captures or loads network-flow features, trains a **Random Forest** model, and outputs a malicious probability.
A third **hybrid layer** consumes both probabilities and applies fixed fusion thresholds.

3. **Training data sources**
The repo uses two static datasets as the main training/evaluation sources:
- `data/raw/syscall/` for syscall modeling, based on **CHIDS**
- `data/raw/network/` for network modeling, based on **Bot-IoT**
These are separate domains, which is important for the architecture because each model learns from a different feature space and data distribution.

4. **Model artifacts**
Trained artifacts are stored under `models/`:
- `models/syscall/best_model.pkl` plus `scaler.pkl`
- `models/network/best_network_model.pkl`
The syscall pipeline depends on a preprocessing scaler, while the network pipeline appears to use a single serialized model artifact.

5. **Runtime capture subsystem**
The live-demo path is in `src/hybrid/runtime/`. It captures telemetry from active Docker containers using:
- **sysdig** for syscall collection
- **tshark** for packet capture
The shell script `run_and_capture.sh` orchestrates capture, and the runtime raw logs/pcaps land under `data/runtime/logs/`.

6. **Runtime feature extraction and schema alignment**
After capture, separate extractors convert raw telemetry into model-ready features:
- `extract_syscall_features.py`
- `extract_network_features.py`
Then alignment scripts normalize the runtime feature schema to match what the trained models expect:
- `align_syscall_schema.py`
- `align_network_schema.py`
This boundary is important: the repo distinguishes **raw capture**, **feature extraction**, and **schema compatibility** as separate steps.

7. **Inference boundary**
Inference is also split by modality:
- `src/syscall/inference/run_syscall_inference.py`
- `src/network/inference/run_network_inference.py`
Each script loads its model artifact and writes prediction CSVs under `outputs/predictions/`. The syscall pipeline also uses the saved scaler before inference.

8. **Hybrid fusion layer**
The fusion subsystem is in `src/hybrid/fusion/run_hybrid_fusion.py`. It joins syscall and network predictions by run identifier and applies a **rule-based late-fusion decision** using probability thresholds rather than a learned meta-model. This means the hybrid result is not a third ML model; it is a deterministic decision layer on top of the two base models.

9. **Evaluation regimes**
There are two distinct evaluation tracks:
- **Synthetic / offline validation**: compares single-modality and hybrid performance on prepared CSVs.
- **Runtime evaluation**: scores the end-to-end live Docker demo scenarios.
These are handled in `src/*/evaluation/` and produce metrics in `outputs/metrics/` and plots in `outputs/plots/`.

10. **Outputs and reporting**
The repo persists artifacts in three main output areas:
- `outputs/predictions/` for per-model and hybrid prediction CSVs
- `outputs/metrics/` for summary metrics tables and text reports
- `outputs/plots/` for confusion matrices, ROC curves, and window-analysis figures
This makes the project reproducible and also supports dissertation-style reporting.

11. **Demo scenarios**
The `demo/` directory contains several prepared runtime scenario scripts that likely drive different container behaviors for live testing. These scenarios feed the runtime capture pipeline and demonstrate how the system behaves on benign and malicious workloads.

12. **Validation and test boundary**
The `tests/` directory contains end-to-end validation scripts for datasets, synthetic evaluations, and hybrid behavior. This indicates the repo is organized around **pipeline correctness across stages**, not just model accuracy.

13. **Key technology choices that shape the architecture**
The important technologies here are:
- **Python 3.11+**
- **scikit-learn** for the classifiers
- **joblib/pickle-style serialization** for model artifacts
- **Docker** as the runtime environment under observation
- **sysdig** and **tshark** as capture tools
These are not incidental; they define how data enters the system and how models are deployed.

14. **Conceptual diagram guidance**
If you are drawing the architecture, the clearest structure is:
**Container workload → capture tools → raw telemetry → feature extraction/schema alignment → modality-specific models → probability outputs → fusion rule → final verdict**, with separate branches for offline training/evaluation and live runtime capture.

Main entry points: [`src/hybrid/fusion/run_hybrid_fusion.py`](https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/hybrid/fusion/run_hybrid_fusion.py) (Late fusion), [`src/syscall/inference/run_syscall_inference.py`](https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/syscall/inference/run_syscall_inference.py) (Syscall infer), [`src/network/inference/run_network_inference.py`](https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/network/inference/run_network_inference.py) (Network infer), [`tests`](https://github.com/mus1afq/hybrid-ids-containers/tree/main/tests) (Validation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data"]
  node_data_syscall[("Syscall data<br/>dataset")]
  node_data_network[("Network data<br/>dataset")]
  node_data_runtime[("Runtime logs<br/>telemetry artifacts")]
end

subgraph group_models["Models"]
  node_model_syscall[("Syscall model<br/>logistic regression artifact")]
  node_model_network[("Network model<br/>random forest artifact")]
end

subgraph group_syscall["Syscall pipeline"]
  node_syscall_train["Syscall train<br/>training"]
  node_syscall_infer["Syscall infer<br/>inference"]
  node_syscall_eval["Syscall eval<br/>evaluation"]
end

subgraph group_network["Network pipeline"]
  node_network_train["Network train<br/>training"]
  node_network_infer["Network infer<br/>inference"]
  node_network_eval["Network eval<br/>evaluation"]
end

subgraph group_hybrid["Hybrid layer"]
  node_capture["Capture<br/>[run_and_capture.sh]"]
  node_extract_align["Feature align<br/>runtime preprocessing"]
  node_fusion["Late fusion<br/>deterministic fusion"]
  node_hybrid_eval["Hybrid eval<br/>evaluation"]
end

subgraph group_outputs["Outputs"]
  node_pred_outputs[("Predictions<br/>artifacts")]
  node_metric_outputs[("Metrics<br/>artifacts")]
  node_plot_outputs[("Plots<br/>artifacts")]
end

subgraph group_validation["Validation"]
  node_demo_scenarios["Demo scenarios<br/>runtime scenarios"]
  node_tests["Validation<br/>test suite"]
end

node_common_config["Config<br/>[config.py]"]

node_common_config -->|"paths"| node_syscall_train
node_common_config -->|"paths"| node_network_train
node_common_config -->|"paths"| node_capture
node_data_syscall -->|"train data"| node_syscall_train
node_data_network -->|"train data"| node_network_train
node_syscall_train -->|"artifact"| node_model_syscall
node_network_train -->|"artifact"| node_model_network
node_model_syscall -->|"load"| node_syscall_infer
node_model_network -->|"load"| node_network_infer
node_data_syscall -->|"validate"| node_syscall_eval
node_data_network -->|"validate"| node_network_eval
node_syscall_infer -->|"writes"| node_pred_outputs
node_network_infer -->|"writes"| node_pred_outputs
node_capture -->|"raw telemetry"| node_data_runtime
node_demo_scenarios -->|"drives"| node_capture
node_data_runtime -->|"input"| node_extract_align
node_extract_align -->|"aligned features"| node_syscall_infer
node_extract_align -->|"aligned features"| node_network_infer
node_syscall_infer -->|"probability"| node_fusion
node_network_infer -->|"probability"| node_fusion
node_fusion -->|"final verdict"| node_hybrid_eval
node_fusion -->|"writes"| node_pred_outputs
node_hybrid_eval -->|"reports"| node_metric_outputs
node_syscall_eval -->|"reports"| node_metric_outputs
node_network_eval -->|"reports"| node_metric_outputs
node_syscall_eval -->|"figures"| node_plot_outputs
node_network_eval -->|"figures"| node_plot_outputs
node_hybrid_eval -->|"figures"| node_plot_outputs
node_tests -->|"checks"| node_data_syscall
node_tests -->|"checks"| node_data_network
node_tests -->|"checks"| node_fusion

click node_common_config "https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/common/config.py"
click node_data_syscall "https://github.com/mus1afq/hybrid-ids-containers/tree/main/data/raw/syscall"
click node_data_network "https://github.com/mus1afq/hybrid-ids-containers/tree/main/data/raw/network"
click node_data_runtime "https://github.com/mus1afq/hybrid-ids-containers/tree/main/data/runtime"
click node_model_syscall "https://github.com/mus1afq/hybrid-ids-containers/tree/main/models/syscall"
click node_model_network "https://github.com/mus1afq/hybrid-ids-containers/tree/main/models/network"
click node_syscall_train "https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/syscall/training/train_syscall_model.py"
click node_syscall_infer "https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/syscall/inference/run_syscall_inference.py"
click node_syscall_eval "https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/syscall/evaluation/evaluate_syscall.py"
click node_network_train "https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/network/training/train_network_model.py"
click node_network_infer "https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/network/inference/run_network_inference.py"
click node_network_eval "https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/network/evaluation/evaluate_network.py"
click node_capture "https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/hybrid/runtime/run_and_capture.sh"
click node_extract_align "https://github.com/mus1afq/hybrid-ids-containers/tree/main/src/hybrid/runtime"
click node_fusion "https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/hybrid/fusion/run_hybrid_fusion.py"
click node_hybrid_eval "https://github.com/mus1afq/hybrid-ids-containers/tree/main/src/hybrid/evaluation"
click node_demo_scenarios "https://github.com/mus1afq/hybrid-ids-containers/tree/main/demo"
click node_tests "https://github.com/mus1afq/hybrid-ids-containers/tree/main/tests"
click node_pred_outputs "https://github.com/mus1afq/hybrid-ids-containers/tree/main/outputs/predictions"
click node_metric_outputs "https://github.com/mus1afq/hybrid-ids-containers/tree/main/outputs/metrics"
click node_plot_outputs "https://github.com/mus1afq/hybrid-ids-containers/tree/main/outputs/plots"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_data_syscall,node_data_network,node_data_runtime,node_demo_scenarios,node_tests toneBlue
class node_model_syscall,node_model_network toneAmber
class node_syscall_train,node_syscall_infer,node_syscall_eval toneMint
class node_network_train,node_network_infer,node_network_eval toneRose
class node_capture,node_extract_align,node_fusion,node_hybrid_eval toneIndigo
class node_pred_outputs,node_metric_outputs,node_plot_outputs toneTeal
class node_common_config toneNeutral
```

## Components

### Data

Static datasets and runtime telemetry artifacts

- [Syscall data](https://github.com/mus1afq/hybrid-ids-containers/tree/main/data/raw/syscall): dataset · CHIDS-derived offline and synthetic syscall datasets · `data/raw/syscall`
- [Network data](https://github.com/mus1afq/hybrid-ids-containers/tree/main/data/raw/network): dataset · Bot-IoT-derived offline and synthetic network datasets · `data/raw/network`
- [Runtime logs](https://github.com/mus1afq/hybrid-ids-containers/tree/main/data/runtime): telemetry artifacts · Captured syscall logs, pcaps, and aligned runtime CSVs · `data/runtime`

### Models

Serialized trained artifacts

- [Syscall model](https://github.com/mus1afq/hybrid-ids-containers/tree/main/models/syscall): logistic regression artifact · Saved syscall classifier plus scaler · `models/syscall`
- [Network model](https://github.com/mus1afq/hybrid-ids-containers/tree/main/models/network): random forest artifact · Saved network classifier artifact · `models/network`

### Syscall pipeline

Syscall-based detection branch

- [Syscall train](https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/syscall/training/train_syscall_model.py): training · Trains the syscall model and scaler · `src/syscall/training/train_syscall_model.py`
- [Syscall infer](https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/syscall/inference/run_syscall_inference.py): inference · Loads syscall artifact and emits prediction CSVs · `src/syscall/inference/run_syscall_inference.py`
- [Syscall eval](https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/syscall/evaluation/evaluate_syscall.py): evaluation · Offline and synthetic syscall scoring · `src/syscall/evaluation/evaluate_syscall.py`

### Network pipeline

Network-flow detection branch

- [Network train](https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/network/training/train_network_model.py): training · Trains the network classifier · `src/network/training/train_network_model.py`
- [Network infer](https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/network/inference/run_network_inference.py): inference · Loads network artifact and emits prediction CSVs · `src/network/inference/run_network_inference.py`
- [Network eval](https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/network/evaluation/evaluate_network.py): evaluation · Offline and synthetic network scoring · `src/network/evaluation/evaluate_network.py`

### Hybrid layer

Runtime capture, feature alignment, and late fusion

- [Capture](https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/hybrid/runtime/run_and_capture.sh): runtime capture · Orchestrates sysdig and tshark collection from Docker workloads · `src/hybrid/runtime/run_and_capture.sh`
- [Feature align](https://github.com/mus1afq/hybrid-ids-containers/tree/main/src/hybrid/runtime): runtime preprocessing · Extracts telemetry features and aligns runtime schemas · `src/hybrid/runtime`
- [Late fusion](https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/hybrid/fusion/run_hybrid_fusion.py): deterministic fusion · Combines modality probabilities with threshold rules · `src/hybrid/fusion/run_hybrid_fusion.py`
- [Hybrid eval](https://github.com/mus1afq/hybrid-ids-containers/tree/main/src/hybrid/evaluation): evaluation · Synthetic and runtime hybrid assessment · `src/hybrid/evaluation`

### Outputs

Predictions, metrics, and plots

- [Predictions](https://github.com/mus1afq/hybrid-ids-containers/tree/main/outputs/predictions): artifacts · Per-model and hybrid prediction CSV outputs · `outputs/predictions`
- [Metrics](https://github.com/mus1afq/hybrid-ids-containers/tree/main/outputs/metrics): artifacts · Summary metrics and report tables · `outputs/metrics`
- [Plots](https://github.com/mus1afq/hybrid-ids-containers/tree/main/outputs/plots): artifacts · Confusion matrices, ROC curves, and analysis figures · `outputs/plots`

### Validation

End-to-end checks and scenario drivers

- [Demo scenarios](https://github.com/mus1afq/hybrid-ids-containers/tree/main/demo): runtime scenarios · Prepared benign and malicious container workloads · `demo`
- [Validation](https://github.com/mus1afq/hybrid-ids-containers/tree/main/tests): test suite · Dataset, synthetic, and end-to-end pipeline checks · `tests`

### Other components

- [Config](https://github.com/mus1afq/hybrid-ids-containers/blob/main/src/common/config.py): shared config · Repository-wide settings and path wiring · `src/common/config.py`

## Connections

- Config → Syscall train: paths · Shared configuration used during training
- Config → Network train: paths · Shared configuration used during training
- Config → Capture: paths · Runtime capture and file locations
- Syscall data → Syscall train: train data · Offline syscall datasets feed model training
- Network data → Network train: train data · Offline network datasets feed model training
- Syscall train → Syscall model: artifact · Produces classifier and scaler artifacts
- Network train → Network model: artifact · Produces trained classifier artifact
- Syscall model → Syscall infer: load · Inference loads saved syscall model and scaler
- Network model → Network infer: load · Inference loads saved network model
- Syscall data → Syscall eval: validate · Offline and synthetic syscall evaluation inputs
- Network data → Network eval: validate · Offline and synthetic network evaluation inputs
- Syscall infer → Predictions: writes · Stores syscall prediction CSVs
- Network infer → Predictions: writes · Stores network prediction CSVs
- Capture → Runtime logs: raw telemetry · Captures logs and packet traces from Docker workloads
- Demo scenarios → Capture: drives · Scenario scripts trigger live container behaviors
- Runtime logs → Feature align: input · Raw runtime telemetry is converted into model-ready features
- Feature align → Syscall infer: aligned features · Aligned syscall runtime features feed inference
- Feature align → Network infer: aligned features · Aligned network runtime features feed inference
- Syscall infer → Late fusion: probability · Supplies syscall malicious probability
- Network infer → Late fusion: probability · Supplies network malicious probability
- Late fusion → Hybrid eval: final verdict · Late-fusion decision is evaluated
- Late fusion → Predictions: writes · Stores hybrid prediction CSVs
- Hybrid eval → Metrics: reports · Produces hybrid metrics tables
- Syscall eval → Metrics: reports · Produces syscall metrics tables
- Network eval → Metrics: reports · Produces network metrics tables
- Syscall eval → Plots: figures · Produces syscall analysis plots
- Network eval → Plots: figures · Produces network analysis plots
- Hybrid eval → Plots: figures · Produces hybrid analysis plots
- Validation → Syscall data: checks · Validates syscall datasets and synthetic cases
- Validation → Network data: checks · Validates network datasets and synthetic cases
- Validation → Late fusion: checks · Validates hybrid behavior and end-to-end flow