# daudibrahimhasan/pokemondatascience architecture

> Architecture diagram of the GitHub repository daudibrahimhasan/pokemondatascience, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/daudibrahimhasan/pokemondatascience
- Repository: https://github.com/daudibrahimhasan/pokemondatascience

## Overview

##### Project Type and Purpose
- This is a standalone Data Science pipeline project organized around three sequential phases:
a. Data Ingestion/Web Scraping
b. Data Preprocessing & Exploratory Data Analysis (EDA)
c. Machine Learning & Computer Vision Modeling
- Its goal is to produce a comprehensive Pokémon dataset, perform cleaning and visual analysis, then build and evaluate several ML/AI models.

##### Main Architectural Components
- External Data Source
- Bulbapedia website (web pages scraped via Python)
- Data Ingestion Layer
- Python web-scraping script (embedded in Jupyter notebook or standalone .py)
- Output: raw “Pokemon.csv”
- Data Storage
- Raw dataset file: Pokemon.csv
- Processed datasets saved within notebook outputs or additional CSVs (implied)
- Data Processing & EDA
- Jupyter notebooks: preprocess.ipynb and visualizationsEDA.ipynb
- Libraries: Pandas, NumPy, Matplotlib, Seaborn, Plotly
- Outputs: cleaned dataset, transformed features, interactive and static visualizations (Images/Graphics/*.png)
- Modeling & Evaluation
- Jupyter notebook: ml.ipynb
- Machine Learning sub-components:
– Classical ML (DecisionTree, RandomForest, GradientBoosting, AdaBoost, LightGBM) for classification/regression
– Unsupervised Clustering (hierarchical + t-SNE)
– Deep Learning CV (Keras with DenseNet, Xception)
- Libraries: scikit-learn, LightGBM, TensorFlow/Keras
- Outputs: model performance metrics, confusion matrices, plots saved under Images/Graphics
- Presentation / Reporting
- README.md for high-level overview
- Notebook outputs (images, tables, interactive charts)

##### Relationships and Data Flow
1. Scraper notebook (ml.ipynb / separate script) fetches HTML → parses JSON/HTML → writes raw CSV
2. preprocess.ipynb reads raw CSV → performs cleaning & feature engineering → writes two artifacts:
a. Cleaned dataset for visualization
b. Cleaned dataset for modeling
3. visualizationsEDA.ipynb reads cleaned-for-EDA dataset → produces static and interactive plots → exports images
4. ml.ipynb reads cleaned-for-modeling dataset → trains & evaluates multiple models → exports metrics and figures
5. All artifacts (CSV, notebooks, generated images) are versioned in Git alongside code

##### Architectural Patterns & Design Principles
- Pipeline pattern: clear linear flow from scraping → cleaning → analysis → modeling
- Separation of concerns:
- Ingestion separated from preprocessing
- EDA separated from model training
- Reproducibility: use of Jupyter notebooks with seed settings, explicit dependencies
- Modularity: each notebook handles a distinct stage, can be replaced or extended independently

##### Technologies & Frameworks
- Python ecosystem: pandas, numpy, requests/BeautifulSoup for scraping
- Visualization: Matplotlib, Seaborn, Plotly
- Machine Learning: scikit-learn, LightGBM
- Deep Learning: TensorFlow / Keras
- Development: Jupyter notebooks
- Version Control: Git, MIT license

##### Diagramming Recommendations
- Labels:
- Component name (e.g., “Web Scraper”)
- Primary output (e.g., “Pokemon.csv”)
- Labeled with data artifact names

##### Detailed Component Descriptions
- External Data Source
- Ingestion Component
- Jupyter Notebook “ml.ipynb” or “scraper.py” (green rectangle)
- Preprocessing Component
- preprocess.ipynb (green rectangle)
- Inputs → Pokemon.csv; Outputs → Cleaned_EDA.csv, Cleaned_ML.csv
- EDA Component
- visualizationsEDA.ipynb (green rectangle)
- Inputs → Cleaned_EDA.csv; Outputs → plot images (Images/Graphics)
- Modeling Component
- ml.ipynb (green rectangle)
- Sub-components:
– Classical ML models → metrics tables
– Clustering → t-SNE plot
– CV models → confusion matrices, accuracy plots
- Outputs → report images, final metrics
- Repository Root

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Data Source
    bulb["Bulbapedia"]:::external

    %% Ingestion
    ingestion["Web Scraper\nml.ipynb\n(pandas, requests)"]:::process
    PokemonCsv[/"Pokemon.csv"/]:::storage

    %% Preprocessing
    preprocessing["Preprocessing\npreprocess.ipynb\n(pandas, numpy)"]:::process
    CleanEDA[/"Cleaned_EDA.csv"/]:::storage
    CleanML[/"Cleaned_ML.csv"/]:::storage

    %% EDA
    eda["EDA & Visualization\nvisualizationsEDA.ipynb\n(Matplotlib, Seaborn, Plotly)"]:::process
    PlotImages(("Plot Images\nImages/Graphics/")):::output

    %% Modeling
    modeling["Modeling & CV\nml.ipynb\n(sklearn, LightGBM, Keras)"]:::process
    Metrics(("Metrics & Figures\nImages/Graphics/")):::output
    spriteIcons(("Sprite Images\nImages/Icons/")):::output
    animatedGifs(("Animated GIFs\nImages/Gifs/")):::output
    miscImages(("Misc Images\nImages/Misc/")):::output

    %% Documentation
    subgraph "Documentation"
        readme([README.md]):::docs
        license([LICENSE]):::docs
    end

    %% Data Flow
    bulb --> ingestion
    ingestion -->|"raw CSV"| PokemonCsv
    PokemonCsv --> preprocessing
    preprocessing -->|"cleaned CSV (EDA)"| CleanEDA
    preprocessing -->|"cleaned CSV (ML)"| CleanML
    CleanEDA --> eda
    eda -->|"plots"| PlotImages
    CleanML --> modeling
    modeling -->|"metrics & figures"| Metrics
    modeling -->|"sprites"| spriteIcons
    modeling -->|"gifs"| animatedGifs
    modeling -->|"misc images"| miscImages

    %% Legend
    subgraph "Legend"
        L1["External\n(Blue)"]:::external
        L2["Storage\n(Orange)"]:::storage
        L3["Process\n(Green)"]:::process
        L4["Output\n(Purple)"]:::output
    end

    %% Click Events
    click ingestion "https://github.com/daudibrahimhasan/pokemondatascience/blob/main/ml.ipynb"
    click PokemonCsv "https://github.com/daudibrahimhasan/pokemondatascience/blob/main/Pokemon.csv"
    click preprocessing "https://github.com/daudibrahimhasan/pokemondatascience/blob/main/preprocess.ipynb"
    click eda "https://github.com/daudibrahimhasan/pokemondatascience/blob/main/visualizationsEDA.ipynb"
    click PlotImages "https://github.com/daudibrahimhasan/pokemondatascience/tree/main/Images/Graphics/"
    click spriteIcons "https://github.com/daudibrahimhasan/pokemondatascience/tree/main/Images/Icons/"
    click animatedGifs "https://github.com/daudibrahimhasan/pokemondatascience/tree/main/Images/Gifs/"
    click miscImages "https://github.com/daudibrahimhasan/pokemondatascience/tree/main/Images/Misc/"
    click readme "https://github.com/daudibrahimhasan/pokemondatascience/blob/main/README.md"
    click license "https://github.com/daudibrahimhasan/pokemondatascience/tree/main/LICENSE"

    %% Styles
    classDef external fill:#AEDFF7,stroke:#0366d6,stroke-width:2px;
    classDef storage fill:#FFD59E,stroke:#d9730d,stroke-width:2px;
    classDef process fill:#C8E6C9,stroke:#2e7d32,stroke-width:2px;
    classDef output fill:#E1BEE7,stroke:#6a1b9a,stroke-width:2px;
    classDef docs fill:#E1BEE7,stroke:#6a1b9a,stroke-width:2px;
```
