How awslabs/sagemaker-corporate-credit-rating works
Project Type and Purpose
• This is an AWS SageMaker JumpStart Industry solution for a _multimodal_ corporate credit rating model. It combines tabular financial data (Altman’s ratios) with text (SEC filings) to train an AutoGluon model, deploy it to a SageMaker endpoint, and serve real-time predictions.
• It also includes CloudFormation templates to provision all AWS infrastructure and a “solution-assistant” Lambda for orchestration or helper workflows.
File Structure Analysis
Top-level directories and files:
• .github & workflows – CI/CD tests, CodeQL scanning, issue templates
• cloudformation – YAML templates for:
– core stack (corporate-credit-rating.yaml)
– nested stacks: endpoint, permissions, notebook instance, SageMaker Studio/solution-assistant Lambda
– Python helper (lambda_fn.py) for solution-assistant
• scripts – packaging and kernel setup for notebooks
• src – two Jupyter notebooks (train & endpoint demo), and Python modules:
– model-training/train.py
– model-inference/inference.py
– prepare_model_code.sh
• README.md – detailed solution overview, architectures, cost/cleanup instructions
• manifest.json, LICENSE, CONTRIBUTING, etc.
Key Architectural Insights from the README
• Data lives in Amazon S3 (synthetic dataset and notebooks).
• Training uses the SageMaker MXNet AutoGluon estimator.
• Deployment produces a real-time HTTPS SageMaker endpoint.
• CloudWatch monitors training jobs and endpoints.
• CloudFormation drives end-to-end provisioning.
• A helper Lambda (“solution-assistant”) can be optionally invoked via AWS events or API Gateway.
Main Components
A. Developer & CI/CD
• GitHub repo & GitHub Actions → triggers tests and validates CloudFormation
B. Infrastructure Provisioning
• AWS CloudFormation (parent stack)
• Nested stacks: IAM Roles, S3 buckets, SageMaker Notebook Instance, Solution Assistant Lambda, SageMaker Endpoint configuration
C. Data Storage
• Amazon S3 buckets: raw synthetic dataset, model artifacts, inference code
D. Compute & Processing
• SageMaker Notebook Instance: runs corporate-credit-rating.ipynb & endpoint-demo.ipynb
• SageMaker Training Job (AutoGluon MXNet) using train.py
• SageMaker Model & Endpoint using inference.py
• (Optional) Solution-Assistant Lambda
E. Monitoring & Logging
• Amazon CloudWatch for logs and metrics of training jobs, endpoint invocations
F. External Clients
• Jupyter notebooks or external applications invoking the SageMaker HTTPS endpoint for predictions
Relationships and Data Flow
1. Developer pushes code → GitHub Actions runs CI → validates infrastructure templates
2. CloudFormation provisions all AWS resources (S3, IAM, Notebook, Lambda, Endpoint config)
3. Data and code land in S3
4. SageMaker Notebook downloads data via SageMaker Python SDK
5. Notebook invokes SageMaker Training Job
– Input: data from S3
– Compute: train.py performs feature engineering & AutoGluon training
– Output: model artifacts to S3
6. Notebook invokes deployment → SageMaker creates Model & Endpoint
– Inference code (inference.py) packaged in container
7. External clients or notebooks send requests to the HTTPS endpoint → receive predictions
8. CloudWatch captures metrics/logs at each stage
9. (Optional) Solution-Assistant Lambda responds to API Gateway or events for helper actions
Architectural Patterns & Principles
• MLOps pipeline: clear separation between data, training, and serving
• Infrastructure as Code: CloudFormation for reproducibility
• Modular design: discrete scripts for training vs inference; nested stacks for isolation
• Serverless compute: SageMaker-managed services & Lambda for orchestration
• Managed storage: S3 for data and artifacts, IAM for least-privilege access
• Monitoring: centralized logging and metrics with CloudWatch