How andreastraut/machine-learning-with-python works
• Data Sources & Ingestion: Use a “Datasets” component to encapsulate various raw datasets (e.g., AirBnB listings, movies CSVs, TF–IDF text files, IoT sensor data). Also note the “iot-environmental-sensor-data-example” folder.
• Data Preprocessing & Pipeline Modules:
– For “Small Data”: Illustrate a preprocessing block that includes data cleaning (handling NaNs, stratified sampling), feature engineering (OneHotEncoding, custom transformers, feature scaling) and pipeline composition (scikit-learn’s Pipeline and ColumnTransformer components).
– For “Big Data”: Include a separate Spark-based preprocessing block that shows operations such as CSV/parquet loading, data cleansing (regex substitution, type casting), and Spark feature transformations (StringIndexer, OneHotEncoder, VectorAssembler).
• Model Training & Evaluation:
– “Small Data” models: DecisionTreeRegressor, LinearRegression, and RandomForestRegressor examples. Show their training, cross-validation (with GridSearchCV and RandomizedSearchCV), evaluation (using RMSE and confidence intervals) and persistence using joblib.
– “Big Data” models: Spark ML’s Linear Regression (and Ridge/Lasso), Decision Tree and clustering (K-Means) within the Dockerized Spark container.
• Experimentation & Visualization: Include components for interactive analysis (Jupyter Notebooks, deepnote links) and visualization modules (e.g., scatter plots, histograms, mind maps) that drive insights.
• Execution Environment:
– Jupyter Notebooks versus IDE (.py files) for “Small Data”, emphasizing separation of concerns.
– A Docker container running Apache Spark for the “Big Data” segment.
• External Integration & Tools: Tag relevant external libraries/frameworks (scikit-learn, Apache Spark ML, Docker) and data sources (Kaggle, public data portals).
Relationships and Data Flow
• From ingestion, the data splits into two parallel pipelines – one designed for “Small Data” (executed in local notebooks/IDE) and the other for “Big Data” (housed in a Docker/Spark container).
• Within each branch, illustrate a sequential flow: Data Cleaning → Feature Engineering (with pipelines) → Model Training → Evaluation/Validation.
• Feedback loops for hyperparameter optimization (GridSearchCV, RandomizedSearchCV) and a connection to persistence (e.g., saving models as .pkl files).
Architectural Patterns and Design Principles
• “pipeline” design pattern in both the scikit-learn and Spark implementations (using sequential transformation and modeling blocks).
• Use of stratified sampling, cross-validation and model selection as vital steps in the machine learning workflow.
• Make it clear how the separation between data preprocessing, model training, and evaluation supports scalability, reproducibility, and clarity.