How aws-samples/ai-ml-sagemaker-multi-model-pipeline works
1. A custom Docker container (in the “container” directory) that implements a Decision Tree algorithm using scikit-learn – complete with training, prediction, serving, and local testing scripts.
2. A SageMaker pipeline (in the “sagemaker-pipeline” directory) that orchestrates a multi-model training strategy. This pipeline supports both the custom Decision Tree (via your custom container) and an XGBoost model (using SageMaker’s built-in container). It brings together preprocessing (using Data Wrangler transformations), hyperparameter tuning, evaluation, and finally registers the winning model for inference deployment.
Main Components
• Container Component (Custom Docker):
• Custom container that includes scripts like “predictor.py”, “train”, “serve”, “wsgi.py”, along with configuration files (e.g., nginx.conf, app-image-config-input.json).
• Local testing space under “local_test” that assists in validating the container functionality.
• SageMaker Pipeline Component:
• Pipeline orchestration that manages both the Decision Tree (custom container) and the XGBoost training.
• Components for preprocessing, hyperparameter tuning, and final evaluation.
• Model registry where the best performing model is stored.
• External Services:
• Connections to AWS services (S3 for data, ECR for container storage, SageMaker for training, and model deployment).
Relationships and Interactions
• Pipeline Flow: After preprocessing, branch the flow into two parallel model training paths:
• One path goes to the custom container (Decision Tree) – indicating that before running, the custom container must be pushed to ECR.
• The other uses the built-in XGBoost container from SageMaker.
• Post-training, both paths converge into an evaluation stage (for hyperparameter tuning and performance comparison) and then move into a model registration component.
• Orchestration: The pipeline scripts (like “run_pipeline.py”, “pipeline.py”, “preprocess.py”, etc.) control the flow.
Architectural Patterns and Technologies
• Containerization is a key element – emphasize Docker usage, Dockerfile structure, and container orchestration.
• The pipeline design follows a modular, microservice-like pattern: distinct encapsulation of preprocessing, training (for two models), evaluation, and registration.
Additional Notes
• That the custom container must first be built and pushed to ECR before being invoked by the pipeline.