How aws-solutions/mlops-workload-orchestrator works
Main Components
• Orchestrator Component: This is deployed using an AWS CloudFormation template. It acts as a central manager (orchestrator account) that handles multiple ML pipelines by integrating with AWS Organizations and deploying pipelines across accounts in multi-account scenarios.
• Pipeline Component: AWS CodePipeline instances are created either via an API Gateway call or by configuration file uploads to an Amazon S3 bucket. These pipelines trigger various ML workflow steps such as training, deploying, and monitoring models.
• AWS Lambda Functions: There is an extensive set of Lambda functions organized under “source/infrastructure/lib/blueprints/lambdas/” and “source/lambdas”. They are responsible for tasks like batch transformation, creating baseline jobs, model training, real-time inference, custom resource invocations, and overall pipeline orchestration.
• Infrastructure as Code (CDK): The “source/infrastructure” folder contains AWS CDK code (including blueprints and pipeline definitions) to deploy the registered ML pipelines and auxiliary components.
• Supporting Configuration and Deployment Scripts: The “deployment” folder contains scripts such as “build-s3-dist.sh” and “run-unit-tests.sh” which are used for packaging, testing, and deploying the solution assets.
Relationships and Interactions
• How API Gateway or S3 event notifications initiate the pipeline creation.
• How the orchestrator leverages AWS Organizations and StackSets to deploy pipelines into multiple AWS accounts or regions.
• How various Lambda functions (grouped by their functionality: custom resource, pipeline orchestration, solution helper) integrate with the overall workflows to complete ML tasks.
Architectural Patterns and Design Principles
• Microservices/Serverless: Emphasize that many tasks are handled by serverless functions (lambdas), which improve scalability and modularity.
• Infrastructure as Code: Use of AWS CDK and CloudFormation templates represents an IaC approach.
• Reusability and Extensibility: The blueprints and pipeline definitions indicate the design is meant to be extended by custom logic (for example, adding new ML pipeline stages).
• Separation of Concerns: Clear isolation between deployment/build scripts, infrastructure definition, and the business logic embedded in the lambda functions.
Relevant Technologies and Frameworks
• AWS CloudFormation and CDK for infrastructure management.
• AWS Lambda for serverless function execution.
• AWS CodePipeline for orchestrating continuous delivery pipelines.
• Amazon SageMaker for model training and deployment.
• AWS Organizations and StackSets for multi-account deployments.
• Docker (for building AWS Lambda layers for custom algorithm images).
• Python as the primary runtime environment and testing frameworks (pytest).
Elements
• Use clear, descriptive labels for each component:
• Orchestrator (CloudFormation/CDK Orchestrator Stack)
• AWS CodePipeline (with single-account and multi-account variants)
• Lambda Function groups (custom resource, pipeline orchestration, solution helper, etc.)
• Supporting components (S3 buckets, API Gateway, Amazon SageMaker endpoints, etc.)
• For example, show flows from the API Gateway/S3 event to the CodePipeline, and then from the pipeline to various Lambda functions.
• How the orchestrator communicates with target accounts in a multi-account setup.
Capturing Essential Architectural Elements
• Modularity. The system is broken down into many smaller, well-organized directories within the repo (each representing a distinct pipeline factor or Lambda service).
• Do not overcomplicate; however, ensure that each architectural layer (deployment scripts, Lambda functions, CDK stack, pipelines, external integrations) is represented as an individual block or container.