How ainilaha/ppl_car_hacking works
Main Components
• Data Source & Ingestion:
• A “data” folder (as per the README instructions) where the Car-Hacking Dataset is saved.
• Data Preprocessing & Conversion:
• A preprocessing step where raw textual data (attack-free traffic) is transformed into CSV format (the README details commands with sed).
• Analysis Modules (Notebooks):
• Notebooks for different parts of the analysis:
• bayesian_net.ipynb (focuses on deep Bayesian network modeling),
• int_data_small.ipynb and bin_data_small.ipynb (likely used for exploring and processing the intrusion/attack data),
• int_pyspark.ipynb (shows Spark-based processing, hinting at handling larger-scale data).
• Utility Module:
• ppl_util.py serves as a helper file with utility functions that support the notebooks.
• Environment/Dependencies:
• Two distinct environment setups: one leveraging TensorFlow (tensorflow-gpu and tensorflow-probability) and another using PyTorch (with pyro-ppl and gpytorch).
Component Relationships & Interactions
• Data Flow:
• Then pass the data through a preprocessing module (including text-to-CSV conversion).
• Analysis Pipeline:
• The processed data is consumed by various notebooks to perform different processing or modeling tasks. They might be executed sequentially or in parallel, depending on the analysis path.
• The "ppl_util.py" acts as a shared resource that provides common functionality to all notebooks.
• Environment Selection:
• Depending on the analysis pipeline chosen (TensorFlow vs PyTorch), the subsequent machine learning model training will utilize one of the two sets of dependencies.
Important Architectural Patterns or Principles
• Modularity:
• Separate processing responsibilities are divided among different notebooks and utility modules.
• Flexibility/Extensibility:
• The design supports multiple ML frameworks with two independent environments, showing extensibility in the choice of backend.
• Data-Centric Flow:
• The architecture follows a straightforward pipeline that begins at data ingestion, continues with preprocessing, and culminates in analytical model building.
Relevant Technologies & Libraries
• Data Processing: Use of Python libraries such as pandas, numpy, matplotlib, and sklearn.
• Deep Learning & Bayesian Inference:
• TensorFlow-based approach with tensorflow-gpu and tensorflow-probability.
• PyTorch-based approach with pytorch, pyro-ppl, and gpytorch.
• Distributed Data Processing: Presence of int_pyspark.ipynb suggests the use of Apache Spark for handling larger datasets.