How datatalksclub/mlops-zoomcamp works
Main Components
• Module Components – Each module (intro, experiment tracking, orchestration, deployment, monitoring, best practices, final project) forms a primary “node” of the learning journey.
• Experiment Tracking – Highlight parts that use MLflow (and even WandB in some cohort solutions).
• Model Deployment – Subdivide this into:
– Batch deployment (with notebooks, Python scripts, and Pipenv dependencies),
– Streaming deployment (including Dockerfiles, AWS Lambda functions, and test scripts),
– Web-service deployment (using Flask, Docker, and standalone prediction scripts).
• Orchestration and Pipelines – Identify orchestration components and notebooks/code that demonstrate workflow management (e.g., Prefect, Mage, or custom orchestration scripts).
• Monitoring – Represent both web service monitoring (with Prometheus, Grafana dashboards, Evidently) and batch job monitoring components.
• Best Practices / CI-CD / Infrastructure – Include components that touch on unit/integration testing, linting, pre-commit hooks, GitHub Actions CI/CD workflows, and the Terraform configurations for infrastructure.
Relationships and Interactions
• From experiment tracking (with model saving/registry) move to “Orchestration/Pipelines,” which would be the process of managing data workflows and training.
• Next, link “Model Deployment,” splitting into the three deployment types (batch, streaming, web-service) where the trained model is put into production.
• The “Best Practices” module can be drawn as an overlay that touches each component (CI/CD, testing, code quality) and supports the entire pipeline.
Architectural Patterns & Technologies
• Modular design – each module encapsulates a stage of the ML lifecycle.
• Microservice-like separation in deployment – distinguish between batch jobs, streaming services, and web API calls.
• CI/CD Integration – include the GitHub workflows (.github/workflows) that trigger tests and deployments.
• Infrastructure as Code – show a side-chain where Terraform scripts (in the best practices module) ensure reproducible deployments.
• Tools/Technologies – Label components with the corresponding technologies (e.g., MLflow, Flask, Docker, AWS Lambda, Prefect, Terraform, Python and Pipfile-based dependency management).
Final Touches
• Use clear, legible labels on every node and connection.