How abdelrhman-abu-alhijaa/ids-ml-project works
1. Project purpose
- This repository implements a machine-learning intrusion detection system (IDS) for the CICIDS2017 network traffic dataset.
- The architecture is split into two clear phases: an offline ML pipeline for data preparation/training, and an online Streamlit dashboard for interactive detection.
2. Primary runtime boundary
- The only user-facing runtime component is app.py, which serves as a Streamlit web app.
- Everything else in the repo is part of the analysis/training workflow: data cleaning, EDA, and model training notebooks.
3. Data source and ingestion
- The system starts from raw CSV network-flow records from CICIDS2017.
- These records represent the external input boundary of the system and are the basis for both training and live-like classification in the dashboard.
4. Data preparation subsystem
- 01_data_cleaing.ipynb is the preprocessing stage.
- Its role is to transform raw traffic data into a cleaned, model-ready dataset, likely handling missing values, feature normalization/selection, and label preparation.
- This stage produces processed data and may also generate intermediate artifacts such as cleaned CSVs and charts.
5. Exploratory analysis subsystem
- 02_eda.ipynb is the analysis/understanding layer.
- It is used to inspect class distribution, feature behavior, and traffic/attack patterns before training.
- This notebook is not part of production inference; it supports dataset understanding and model design decisions.
6. Model training and evaluation subsystem
- 03_model_training.ipynb is the training pipeline.
- It compares multiple classifiers: Random Forest, XGBoost, and Decision Tree.
- The notebook evaluates models using metrics like Accuracy, F1, and ROC-AUC, and the README indicates the selected/validated model reaches perfect scores on the reported dataset.
7. Model artifact boundary
- Trained model files are intended to be stored under a models/ directory, even though the tree shown does not list the folder explicitly.
- This is the bridge between offline training and online inference: the dashboard consumes the saved model artifact rather than retraining on startup.
8. Interactive detection subsystem
- The Streamlit dashboard in app.py is the operational interface for detection.
- It accepts traffic-like input, runs the trained classifier, and returns:
- predicted attack/benign class,
- confidence score,
- and likely a user-visible alert state.
9. Alert/export subsystem
- The app includes an export path for detection alerts as CSV.
- This means the dashboard is not only a visualizer; it also produces persisted outputs for downstream review or incident response.
10. Model capability and classification scope
- The system is positioned as a multi-class IDS, detecting 13 attack types in addition to benign traffic.
- That makes the core prediction boundary a multi-class classification service, not binary anomaly detection.
11. Technology stack boundaries
- Python is the implementation language across notebooks and app.
- Core libraries define the architecture:
- pandas / numpy for data handling,
- matplotlib / seaborn for analysis and chart generation,
- scikit-learn / xgboost for model training and evaluation,
- Streamlit for the front-end/runtime dashboard,
- joblib for model persistence/loading.
12. Repository structure and artifact flow
- The intended flow is:
- raw dataset → cleaning notebook → processed data/charts → EDA notebook → training notebook → saved model(s) → Streamlit app.
- The report.md file appears to capture the written project report, while the notebooks contain the executable research/training workflow.
13. What is not in this repo
- There is no evidence of a separate backend API, message queue, database, or streaming ingestion pipeline.
- The architecture is therefore a single-process dashboard plus offline notebooks, rather than a distributed production IDS stack.
14. How to diagram it
- Draw four main blocks:
- External dataset input (CICIDS2017)
- Offline ML pipeline (cleaning, EDA, training, model persistence)
- Saved model artifact store
- Streamlit dashboard (interactive inference, confidence display, CSV export)
- The key dependency arrow is from trained model artifact → Streamlit app, and from raw data → notebooks → trained model.
Main entry points: 01_data_cleaing.ipynb Data Cleaning · 03_model_training.ipynb Model Training · report.md Report · app.py Dashboard
Components 16
External Data
Raw CICIDS2017 network-flow records entering the project
- CICIDS2017 · Source network-traffic CSV records used for training and dashboard inputs
Offline ML Pipeline
Notebook-driven preparation, analysis, training, and model packaging
- Data Cleaning · Cleans raw traffic data and prepares model-ready records
01_data_cleaing.ipynb - Feature Prep · Representative preprocessing step for normalization, encoding, and label shaping
01_data_cleaing.ipynb - EDA · Explores class balance, feature behavior, and attack patterns
02_eda.ipynb - Class Study · Representative inspection of traffic classes and attack distributions
02_eda.ipynb - Model Training · Trains and compares Random Forest, XGBoost, and Decision Tree models
03_model_training.ipynb - Model Compare · Representative classifier benchmarking and selection step
03_model_training.ipynb - Metrics · Accuracy, F1, and ROC-AUC evaluation for candidate models
03_model_training.ipynb - Saved Model · Persisted trained model artifact consumed by the dashboard
Streamlit Runtime
Interactive detection and alert export in the only user-facing app
- Dashboard · User-facing Streamlit interface for interactive intrusion detection
app.py - Input Form · Collects traffic-like feature values from the user
app.py - Inference · Loads the saved classifier and predicts benign or attack class
app.py - Confidence · Displays prediction confidence and alert state
app.py - CSV Export · Exports detection alerts for review or incident response
app.py
Connections 18
- CICIDS2017 → Data Cleaning: raw flows · Raw CSV traffic records are cleaned for downstream analysis
- Data Cleaning → Feature Prep: preprocess · Cleaning produces model-ready feature and label data
- Feature Prep → EDA: prepared data · Prepared data is inspected in exploratory analysis
- EDA → Class Study: inspect · EDA surfaces class distribution and traffic patterns
- Class Study → Model Training: findings · Analysis informs model training choices
- Data Cleaning → Model Training: dataset · Processed data feeds the training notebook
- Model Training → Model Compare: benchmark · Multiple classifiers are compared during training
- Model Compare → Metrics: score · Candidate models are evaluated with classification metrics
- Metrics → Saved Model: persist best · Selected trained model is saved for inference
- Saved Model → Dashboard: load model · The Streamlit app loads the persisted artifact at runtime
- Dashboard → Input Form: collect input · The dashboard captures traffic-like feature values
- Input Form → Inference: features · Entered values are passed to the inference step
- Inference → Confidence: predict · Inference returns class and confidence information
- Confidence → CSV Export: alert record · Predictions can be exported as CSV alerts
- Report → README: document · Project write-up complements repository usage notes
- Data Cleaning → Report: results · Preprocessing and experiment outcomes are summarized in the report
- Model Training → Report: results · Training and evaluation results are captured in the report
- Dashboard → Report: behavior · The report documents the dashboard and detection workflow