# andreastraut/machine-learning-with-python architecture

> Architecture diagram of the GitHub repository andreastraut/machine-learning-with-python, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/andreastraut/machine-learning-with-python
- Repository: https://github.com/andreastraut/machine-learning-with-python

## Overview

- Data Sources & Ingestion: Use a “Datasets” component to encapsulate various raw datasets (e.g., AirBnB listings, movies CSVs, TF–IDF text files, IoT sensor data). Also note the “iot-environmental-sensor-data-example” folder.
- Data Preprocessing & Pipeline Modules:
– For “Small Data”: Illustrate a preprocessing block that includes data cleaning (handling NaNs, stratified sampling), feature engineering (OneHotEncoding, custom transformers, feature scaling) and pipeline composition (scikit-learn’s Pipeline and ColumnTransformer components).
– For “Big Data”: Include a separate Spark-based preprocessing block that shows operations such as CSV/parquet loading, data cleansing (regex substitution, type casting), and Spark feature transformations (StringIndexer, OneHotEncoder, VectorAssembler).
- Model Training & Evaluation:
– “Small Data” models: DecisionTreeRegressor, LinearRegression, and RandomForestRegressor examples. Show their training, cross-validation (with GridSearchCV and RandomizedSearchCV), evaluation (using RMSE and confidence intervals) and persistence using joblib.
– “Big Data” models: Spark ML’s Linear Regression (and Ridge/Lasso), Decision Tree and clustering (K-Means) within the Dockerized Spark container.
- Experimentation & Visualization: Include components for interactive analysis (Jupyter Notebooks, deepnote links) and visualization modules (e.g., scatter plots, histograms, mind maps) that drive insights.
- Execution Environment:
– Jupyter Notebooks versus IDE (.py files) for “Small Data”, emphasizing separation of concerns.
– A Docker container running Apache Spark for the “Big Data” segment.
- External Integration & Tools: Tag relevant external libraries/frameworks (scikit-learn, Apache Spark ML, Docker) and data sources (Kaggle, public data portals).

##### Relationships and Data Flow
- From ingestion, the data splits into two parallel pipelines – one designed for “Small Data” (executed in local notebooks/IDE) and the other for “Big Data” (housed in a Docker/Spark container).
- Within each branch, illustrate a sequential flow: Data Cleaning → Feature Engineering (with pipelines) → Model Training → Evaluation/Validation.
- Feedback loops for hyperparameter optimization (GridSearchCV, RandomizedSearchCV) and a connection to persistence (e.g., saving models as .pkl files).

##### Architectural Patterns and Design Principles
- “pipeline” design pattern in both the scikit-learn and Spark implementations (using sequential transformation and modeling blocks).
- Use of stratified sampling, cross-validation and model selection as vital steps in the machine learning workflow.
- Make it clear how the separation between data preprocessing, model training, and evaluation supports scalability, reproducibility, and clarity.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion Layer
    subgraph "Data Ingestion"
        Datasets["Raw Datasets"]:::ingestion
        subgraph "Datasets Sources"
            AB["AirBnB"]:::ingestion
            TF["TF–idf"]:::ingestion
            ENV["Environmental Sensor Data"]:::ingestion
            MOV["Movies"]:::ingestion
            IoT["IoT Sensor Data"]:::ingestion
        end
    end

    %% Small Data Pipeline
    subgraph "Small Data Pipeline"
        SD_Preproc["Data Cleaning & Feature Engineering (scikit-learn)"]:::smallData
        NB1["Movies Machine Learning - StratifiedSample.ipynb"]:::smallData
        NB2["Movies Machine Learning - Predict NaNs.ipynb"]:::smallData
        PY["Sklearn Machine-Learning_AirBnB.py"]:::smallData
    end

    %% Big Data Pipeline
    subgraph "Big Data Pipeline"
        BD_Preproc["Data Loading & Spark Feature Transformations"]:::bigData
        Docker["Docker: Apache Spark ML Environment"]:::bigData
    end

    %% Model Training & Evaluation
    subgraph "Model Training & Evaluation"
        SD_Model["Model Training (Small Data Models)"]:::model
        SD_Model1["forest_reg.pkl"]:::model
        SD_Model2["my_model.pkl"]:::model
        IoT_Model["IoT Sensor Example Model"]:::model
        IoT_Model1["myModel.pkl"]:::model
        IoT_Model2["predictions.csv"]:::model
        BD_Model["Model Training (Big Data Models)"]:::model
        Eval["Evaluation & Persistence"]:::model
    end

    %% Experimentation & Visualization
    subgraph "Experimentation & Visualization"
        Interactive["iot-environmental-sensor-data-example/iot-environmental-sensor-data_ATraut.ipynb"]:::experiment
        VisualMedia["Media: Data Scientist - Python Jupyter SkLearn.html"]:::experiment
    end

    %% External Integration & Documentation
    subgraph "External & Documentation"
        PDF["Machine-Learning-with-Python_AndreasTraut.pdf"]:::external
        Gitignore[".gitignore"]:::external
        README["README.md"]:::external
        Jovian[".jovianrc"]:::external
        LICENSE["LICENSE"]:::external
    end

    %% Connections from Data Ingestion to Pipelines
    Datasets -->|"feeds"| SD_Preproc
    Datasets -->|"feeds"| BD_Preproc
    AB --> Datasets
    TF --> Datasets
    ENV --> Datasets
    MOV --> Datasets
    IoT --> Datasets

    %% Connections within Small Data Pipeline
    NB1 -->|"provides"| SD_Preproc
    NB2 -->|"provides"| SD_Preproc
    PY  -->|"provides"| SD_Preproc
    SD_Preproc -->|"transforms"| SD_Model

    %% Connections for Big Data Pipeline
    BD_Preproc -->|"transforms"| BD_Model
    Docker --> BD_Preproc

    %% IoT specific flow from ingestion
    IoT -->|"supplies"| IoT_Model
    IoT_Model -->|"trains"| IoT_Model1
    IoT_Model -->|"trains"| IoT_Model2

    %% Model Training & Evaluation Flow
    SD_Model -->|"saves"| SD_Model1
    SD_Model -->|"saves"| SD_Model2
    SD_Model --> Eval
    BD_Model --> Eval
    IoT_Model --> Eval
    Eval -->|"GridSearchCV_RandomizedSearchCV"| SD_Model

    %% Experimentation connections
    Eval -->|"analyzedBy"| Interactive
    Eval -->|"visualizedBy"| VisualMedia

    %% External Documentation connections
    PDF -->|"references"| Eval
    Gitignore ---|"docs"| Eval
    README ---|"docs"| Eval
    Jovian ---|"docs"| Eval
    LICENSE ---|"docs"| Eval

    %% Click Events for Data Ingestion components
    click AB "https://github.com/andreastraut/machine-learning-with-python/tree/master/datasets/AirBnB"
    click TF "https://github.com/andreastraut/machine-learning-with-python/tree/master/datasets/TF-idf"
    click ENV "https://github.com/andreastraut/machine-learning-with-python/tree/master/datasets/environmental-sensor-data-132k"
    click MOV "https://github.com/andreastraut/machine-learning-with-python/tree/master/datasets/movies"
    click IoT "https://github.com/andreastraut/machine-learning-with-python/tree/master/iot-environmental-sensor-data-example"

    %% Click Events for Small Data Pipeline components
    click NB1 "https://github.com/andreastraut/machine-learning-with-python/blob/master/Movies Machine Learning - StratifiedSample.ipynb"
    click NB2 "https://github.com/andreastraut/machine-learning-with-python/blob/master/Movies Machine Learning - Predict NaNs.ipynb"
    click PY "https://github.com/andreastraut/machine-learning-with-python/blob/master/Sklearn Machine-Learning_AirBnB.py"

    %% Click Events for Model Training components (Small Data)
    click SD_Model1 "https://github.com/andreastraut/machine-learning-with-python/blob/master/forest_reg.pkl"
    click SD_Model2 "https://github.com/andreastraut/machine-learning-with-python/blob/master/my_model.pkl"
    %% Click Events for IoT Model Training components
    click IoT_Model1 "https://github.com/andreastraut/machine-learning-with-python/blob/master/iot-environmental-sensor-data-example/myModel.pkl"
    click IoT_Model2 "https://github.com/andreastraut/machine-learning-with-python/blob/master/iot-environmental-sensor-data-example/predictions.csv"

    %% Click Events for Experimentation & Visualization components
    click Interactive "https://github.com/andreastraut/machine-learning-with-python/blob/master/iot-environmental-sensor-data-example/iot-environmental-sensor-data_ATraut.ipynb"
    click VisualMedia "https://github.com/andreastraut/machine-learning-with-python/blob/master/media/Data Scientist - Python  Jupyter SkLearn.html"

    %% Click Events for External & Documentation components
    click PDF "https://github.com/andreastraut/machine-learning-with-python/blob/master/Machine-Learning-with-Python_AndreasTraut.pdf"
    click Gitignore "https://github.com/andreastraut/machine-learning-with-python/blob/master/.gitignore"
    click README "https://github.com/andreastraut/machine-learning-with-python/blob/master/README.md"
    click Jovian "https://github.com/andreastraut/machine-learning-with-python/blob/master/.jovianrc"
    click LICENSE "https://github.com/andreastraut/machine-learning-with-python/tree/master/LICENSE"

    %% Styles
    classDef ingestion fill:#87CEFA,stroke:#1E90FF,stroke-width:2px;
    classDef smallData fill:#90EE90,stroke:#228B22,stroke-width:2px;
    classDef bigData fill:#F4A460,stroke:#CD853F,stroke-width:2px;
    classDef model fill:#FFA500,stroke:#FF8C00,stroke-width:2px;
    classDef experiment fill:#DB7093,stroke:#C71585,stroke-width:2px;
    classDef external fill:#9370DB,stroke:#663399,stroke-width:2px;
```
