How gustavoeso/phishingurl-predictive-model works
Project Type and Purpose
• The project is a machine learning experiment to detect phishing URLs.
• Its main function is to extract features from URLs, train a predictive model (using techniques like Gradient Boosting Trees), and generate predictions with transparent probability outputs.
• The project makes use of key technologies like Python, PySpark, Databricks, and MLflow, which hint at a distributed data processing and experiment tracking workflow.
File Structure
• The project consists of a single Jupyter Notebook (Projeto_Modelo_Preditivo_Phishing_final.ipynb) alongside the README. This indicates that the main logic of the process—data ingestion, feature extraction, model training, and evaluation—is implemented in this notebook.
• The minimal file structure implies that there are no separate directories for frontend/backend or multiple modules; instead, the entire workflow is bundled into this notebook, common in experimental or academic projects.
README for Additional Insights
• The README clearly outlines the project’s purpose (phishing URL detection) and the ML tasks involved.
• It describes the technologies used (Python, PySpark, Databricks, MLflow) and provides an overview of the functionalities like feature extraction, training and validation, and detailed prediction probabilities.
• It references the Kaggle dataset “PhiUSIIL Phishing URL Dataset” as the source, which is a key external dependency.
a. Main Components to Include:
• Data Source: Indicate the external dataset (Kaggle – PhiUSIIL Phishing URL Dataset).
• Data Ingestion & Preprocessing: A module that handles reading the dataset and extracting relevant features (URL length, character continuity, TLD legitimacy probability, etc.).
• Feature Extraction Module: Transform raw URL data into features that the model can utilize.
• Model Training Pipeline: Show where the Gradient Boosting Trees (GBT) model is trained using PySpark.
• Experiment Tracking: Include MLflow to indicate the tracking and monitoring of different training runs.
• Execution Environment: Represent the usage of Databricks, highlighting that the notebook is run on a cloud-based distributed processing platform.
• Prediction/Inference Module: Illustrate the component that applies the trained model to generate predictions with associated probabilities.
b. Relationships and Interactions:
c. Highlighting Architectural Patterns/Design Principles:
• A pipeline or workflow design pattern where data flows sequentially through discrete processing stages.
• Modularity of tasks—each module (feature extraction, training, inference) can be represented as a separate block.
• Mention the use of distributed data processing with PySpark and a cloud-based execution environment at Databricks.
• That MLflow serves as a monitoring and logging layer, which is an essential design principle for reproducibility and experiment tracking.
d. Including Relevant Technologies & Frameworks:
Elements
• Each module/component must have a clear label.