How matheusham/minimo_mlops works
Project type and purpose
• This is a minimal MLOps demo that combines data engineering, model training, and model serving in one Docker‐ized service.
• The goal: consume a CSV of credit‐card transactions, build a “revenge spending” classifier, persist the model, and expose a REST API for predictions.
File structure
• Top level:
• Dockerfile / Makefile / requirements.txt → container build & execution
• assets/ → raw input data (MiBolsillo.csv)
• data_engineering/ → ETL scripts & exploratory notebook
• model_pipeline/ → training pipeline script & exploration notebook
• model_serving/ → FastAPI app (main.py) + Makefile
• No external microservices—everything lives in one container, but logically it’s three stages.
README for extra insight
• Describes a minimal MLOps ecosystem: data ingestion → feature engineering → training → evaluation → serving → monitoring.
• Mentions technologies: Python, PySpark (hinted), scikit‐learn, FastAPI, Docker, optionally DVC/MLflow.
1) Data Source
• CSV file (assets/MiBolsillo.csv)
2) Data Engineering Stage
• prod_data_eng_pipeline.py (pyspark or pandas transforms)
3) Model Training Stage
• prod_model_pipeline.py (scikit‐learn pipeline, hyperparam tuning, model serialization)
4) Model Serving Stage
• FastAPI app (main.py), /predict endpoint
5) Docker Container
• Builds image, runs the three stages at container startup, then hosts FastAPI
6) Client
• Browser or HTTP client hitting the Swagger UI /docs and the prediction API
b. Relationships & data flow:
• Surround Data Eng, Training, Serving with a “Docker Container” boundary
c. Architectural patterns & design principles:
• Pipeline pattern: sequential ETL → train → serve
• Single‐container MLOps demo (for simplicity)
• Separation of concerns: three logical modules in code
• Potential extension points: swap out data source for streaming, external feature store, separate microservices
d. Key technologies / libraries:
• Python 3.x
• PySpark/Pandas (data_engineering)
• scikit‐learn (model_pipeline)
• FastAPI + Uvicorn (model_serving)
• Docker + Make (or direct docker CLI)
• Optionally DVC/MLflow for model versioning (mentioned in README)
Extra details
• The Dockerfile as the orchestration mechanism: annotate that it builds the Python environment, installs dependencies, then runs data_eng → model_train → uvicorn main:app.
• Mention the Makefile targets (make build, make run) as convenience scripts.
• Where model artifacts live (e.g., /app/model.pkl inside container).
• Data Ingestion → Feature Engineering → Model Training → Model Serving
• All steps containerized and orchestrated by Docker/Make
• Client interaction via REST API