How gokumohandas/mlops-course works
Main Components
• Data Management: This includes the datasets (CSV files in the “datasets” directory) and any utilities for data loading (see madewithml/data.py) plus tests in tests/data.
• Core ML Workloads: The “madewithml” folder contains key scripts for training (train.py), tuning (tune.py), evaluation (evaluate.py), inference/prediction (predict.py) and serving (serve.py) as well as supporting modules (config.py, models.py, utils.py). Also note that “notebooks” provide an interactive environment for exploring these functions.
• Experiment Tracking: The README covers the integration with MLflow for tracking experiments and model registry setup.
• Deployment Infrastructure: The “deploy” directory holds configuration files and scripts (cluster compute/environment descriptions, job definitions, service definitions) for deploying workloads and model serving.
• CI/CD & Automation: The “.github/workflows” files, pre-commit hooks, Makefile, and other configuration files enable automated testing, job submission, and service rollout via GitHub Actions.
Relationships and Interactions
• A clear flow exists from data ingestion (using CSVs) → model training (train.py) → hyperparameter tuning (tune.py) → evaluation (evaluate.py) and inference (predict.py) → model serving (serve.py).
• The MLflow tracking system is used to record experiments that are produced across training, tuning, and evaluation workflows.
• Deployment configurations (in deploy/) use Anyscale and Ray to run jobs and serve models in production.
• CI/CD workflows (in “.github/workflows”) trigger jobs (using workloads/job configurations) and serve deployments when merging into the main branch.
Important Architectural Patterns and Design Principles
• Modularity: Each major ML function is implemented as a separate Python script/module. This modularity allows for independent testing (see tests/code, tests/data, and tests/model) and easy extension.
• Separation of Concerns in CI/CD: There’s a clear separation between the development (local notebooks and scripts) and production configurations (deployment YAML files and CI/CD workflows).
• Scalability via Cluster Configuration: The use of cluster environments, compute configs (deploy/cluster_compute.yaml, deploy/cluster_env.yaml) and orchestration of Anyscale jobs/services indicates emphasis on scaling ML workloads.
• Experiment Reproducibility: Usage of MLflow and containerized configurations ensures reproducibility of results across environments.
Relevant Technologies, Frameworks, and Tools
• Python (with modular scripts for each ML stage).
• Jupyter Notebooks for experimentation.
• MLflow for experiment tracking.
• Ray for distributed model serving and workload management.
• Anyscale for cluster management and compute orchestration.
• GitHub Actions, pre-commit tooling, and Docker-like deployment configs (YAML files in deploy) for CI/CD integration.
• Pytest is used for testing various components.
Putting It All Together
• A section for “Deployment & Orchestration” that details both CI/CD (GitHub Actions workflows, pre-commit) and cluster management components (deploy/ directory, Anyscale & Ray orchestration).