How daudibrahimhasan/pokemondatascience works
Project Type and Purpose
• This is a standalone Data Science pipeline project organized around three sequential phases:
a. Data Ingestion/Web Scraping
b. Data Preprocessing & Exploratory Data Analysis (EDA)
c. Machine Learning & Computer Vision Modeling
• Its goal is to produce a comprehensive Pokémon dataset, perform cleaning and visual analysis, then build and evaluate several ML/AI models.
Main Architectural Components
• External Data Source
• Bulbapedia website (web pages scraped via Python)
• Data Ingestion Layer
• Python web-scraping script (embedded in Jupyter notebook or standalone .py)
• Output: raw “Pokemon.csv”
• Data Storage
• Raw dataset file: Pokemon.csv
• Processed datasets saved within notebook outputs or additional CSVs (implied)
• Data Processing & EDA
• Jupyter notebooks: preprocess.ipynb and visualizationsEDA.ipynb
• Libraries: Pandas, NumPy, Matplotlib, Seaborn, Plotly
• Outputs: cleaned dataset, transformed features, interactive and static visualizations (Images/Graphics/*.png)
• Modeling & Evaluation
• Jupyter notebook: ml.ipynb
• Machine Learning sub-components:
– Classical ML (DecisionTree, RandomForest, GradientBoosting, AdaBoost, LightGBM) for classification/regression
– Unsupervised Clustering (hierarchical + t-SNE)
– Deep Learning CV (Keras with DenseNet, Xception)
• Libraries: scikit-learn, LightGBM, TensorFlow/Keras
• Outputs: model performance metrics, confusion matrices, plots saved under Images/Graphics
• Presentation / Reporting
• README.md for high-level overview
• Notebook outputs (images, tables, interactive charts)
Relationships and Data Flow
1. Scraper notebook (ml.ipynb / separate script) fetches HTML → parses JSON/HTML → writes raw CSV
2. preprocess.ipynb reads raw CSV → performs cleaning & feature engineering → writes two artifacts:
a. Cleaned dataset for visualization
b. Cleaned dataset for modeling
3. visualizationsEDA.ipynb reads cleaned-for-EDA dataset → produces static and interactive plots → exports images
4. ml.ipynb reads cleaned-for-modeling dataset → trains & evaluates multiple models → exports metrics and figures
5. All artifacts (CSV, notebooks, generated images) are versioned in Git alongside code
Architectural Patterns & Design Principles
• Pipeline pattern: clear linear flow from scraping → cleaning → analysis → modeling
• Separation of concerns:
• Ingestion separated from preprocessing
• EDA separated from model training
• Reproducibility: use of Jupyter notebooks with seed settings, explicit dependencies
• Modularity: each notebook handles a distinct stage, can be replaced or extended independently
Technologies & Frameworks
• Python ecosystem: pandas, numpy, requests/BeautifulSoup for scraping
• Visualization: Matplotlib, Seaborn, Plotly
• Machine Learning: scikit-learn, LightGBM
• Deep Learning: TensorFlow / Keras
• Development: Jupyter notebooks
• Version Control: Git, MIT license
Diagramming Recommendations
• Labels:
• Component name (e.g., “Web Scraper”)
• Primary output (e.g., “Pokemon.csv”)
• Labeled with data artifact names
Detailed Component Descriptions
• External Data Source
• Ingestion Component
• Jupyter Notebook “ml.ipynb” or “scraper.py” (green rectangle)
• Preprocessing Component
• preprocess.ipynb (green rectangle)
• Inputs → Pokemon.csv; Outputs → Cleaned_EDA.csv, Cleaned_ML.csv
• EDA Component
• visualizationsEDA.ipynb (green rectangle)
• Inputs → Cleaned_EDA.csv; Outputs → plot images (Images/Graphics)
• Modeling Component
• ml.ipynb (green rectangle)
• Sub-components:
– Classical ML models → metrics tables
– Clustering → t-SNE plot
– CV models → confusion matrices, accuracy plots
• Outputs → report images, final metrics
• Repository Root