How tanish-sarkar/network-intrusion-detection-system works
Project type and purpose
• This is a data‐science/ML pipeline project for unsupervised learning on high-dimensional data.
• It implements clustering (K-Means, DBSCAN, Agglomerative), dimensionality reduction (PCA, t-SNE, UMAP), and anomaly detection (Isolation Forest, LOF).
File structure
• Top level:
• data/ (raw and processed CSVs)
• src/ (modular code: data_loader.py, preprocessing.py, dimensionality.py, clustering.py, anomaly.py, utils.py)
• notebooks/ (Jupyter notebooks for EDA, clustering, dimensionality reduction, anomaly detection)
• outputs/ (saved models in joblib, plots)
• main.py (pipeline orchestrator)
• requirements.txt lists dependencies (Pandas, NumPy, scikit-learn, Matplotlib/Seaborn/Plotly).
• README.rst describes project goals, techniques, dataset, and outcome.
README for architecture insights
• Modular Python codebase with clear separation: data processing, dimensionality reduction, clustering, anomaly detection.
• Emphasis on reproducibility and scalability.
1. Data Sources
• Raw files (Train_data.csv, Test_data.csv)
2. Data Loader (src/data_loader.py)
3. Preprocessing (src/preprocessing.py)
• Cleaning, scaling, encoding
4. Dimensionality Reduction (src/dimensionality.py)
• PCA, t-SNE, UMAP
5. Clustering Module (src/clustering.py)
• KMeans, DBSCAN, Agglomerative
6. Anomaly Detection Module (src/anomaly.py)
• Isolation Forest, LOF
7. Model Persistence (joblib via src/utils.py)
8. Visualization / Reporting
• Matplotlib/Seaborn/Plotly & notebooks
9. Notebooks Layer (notebooks/*.ipynb) for EDA and interactive analysis
10. Outputs (models/, plots/)
• Raw Data → Data Loader → Preprocessing →
→ [branch] → Dimensionality Reduction → Clustering → Model Persistence → Outputs/models
→ Anomaly Detection → Model Persistence → Outputs/models
• Preprocessed & Reduced data → Visualization → Outputs/plots
• Notebooks import modules from src to reproduce each step interactively.
• main.py orchestrates end-to-end pipeline by calling each component sequentially.
c. Architectural patterns & principles:
• Pipeline pattern: each stage is a standalone module; data flows in a linear sequence with branches for clustering vs. anomaly detection.
• Separation of concerns: data I/O, preprocessing, feature transformation, modeling, persistence, and visualization are decoupled.
• Reproducibility: main.py + modular code + notebooks + outputs directory.
d. Key technologies & frameworks:
• Python modules in src/
• Pandas, NumPy for data manipulation
• Scikit-learn for algorithms
• Joblib (via src/utils.py) for model persistence
• Matplotlib/Seaborn/Plotly for plotting
• Jupyter Notebooks for exploration
Key points to capture
• Both batch pipeline (main.py) and interactive notebooks.
• Model training vs. model saving vs. visualization.
• Keep it detailed but focused on the true flow of data and responsibilities.