How vincentschaik/production works
Project type and purpose
• This is an educational “ML Systems Design” module repository rather than a customer-facing application.
• It bundles slide decks and Jupyter labs (folders 01_materials, 02_activities, 03_instructional_team, 04_this_cohort) plus a demo source tree (05_src) that spins up a mini production-like experiment-tracking backend via Docker.
File structure
• Top level:
• .github: PR/issue templates and workflows
• 01_materials: Live-lab notebooks & PDF slides
• 02_activities: Graded assignments & rubrics
• 03_instructional_team: Markdown slides + slide-generator script
• 04_this_cohort: Cohort-specific extras
• 05_src: Demo code + infra for ML pipelines
• Inside 05_src:
• credit/: data ingestion and three logistic-regression experiments
• stock_prices/: data_manager for another demo
• experiment_tracking/: docker-compose setup for MLflow, MinIO, PostgreSQL + test script
• utils/: shared logging utility
• Config: environment.yaml, .env files, docker-compose.yml, Dockerfiles, SQL init scripts.
README for additional insights
• The module covers data engineering, feature engineering, hyperparameter tuning, deployment, explainability, experiment tracking/monitoring.
• Students will run Docker to mimic production (Postgres + S3-style storage via MinIO) and use MLflow for experiment tracking.
a. Main components
• Learner’s client: VSCode/Jupyter (interactive notebooks)
• Code modules:
– Data ingestion (credit/data.py, stock_prices/data_manager.py)
– Experiment scripts (exp__*.py)
– Utilities (logger.py)
• ML pipeline stages: data → features → training → tracking → explainability
• Experiment-tracking backend (Docker Compose):
– MLflow Tracking Server
– PostgreSQL metadata store
– MinIO artifact store
• Instructional content (slides, assignments) – static
b. Relationships and data flows
1. Learner opens a Jupyter notebook → invokes data ingestion modules → yields raw data
2. Notebook or script runs pipeline/experiments → calls MLflow Python client
3. MLflow client writes metrics to Tracking Server
5. Artifacts (models, plots) pushed to MinIO buckets
6. Student reads results back in notebook for explainability and monitoring demos
c. Architectural patterns & design principles
• Modularity: separate folders for content, code demos, infra
• Infrastructure as code: docker-compose defines a small microservice ecosystem
• Clear separation of concerns:
– Data ingestion vs. model training vs. logging/tracking
• Reproducibility: Environment specs (environment.yaml, .env) and containerization
d. Key technologies & frameworks
• Python + Jupyter Notebooks
• MLflow (experiment tracking)
• MinIO (S3-compatible object storage)
• PostgreSQL (tracking DB)
• Docker & Docker Compose
• VSCode (Jupyter + Python extensions)