How aws-samples/attribute-amazon-emr-costs-to-your-end-users works
Project type and purpose
• This is an AWS‑focused, open‑source cost‑attribution solution rather than a traditional full‑stack web app.
• It uses AWS CDK (in Python) to deploy a serverless extraction pipeline: a Lambda function that pulls metrics and cost data from EMR, EC2 and Cost Explorer, writes to a Postgres RDS data model, and surfaces dashboards in QuickSight.
File structure
• Top level:
• app.py – CDK entry point
• cdk.json + cdk.context.json – CDK configuration
• cdk_emrcostblog/ – CDK stack code (cdk_emrcostblog_stack.py)
• Lambda/ – business‑logic code (lambda_function.py)
• emr-cost-rds-tables-ddl.sql + sample-cost-calculation-query.sql – data model and example queries
• requirements*.txt – Python dependencies
• tests/ – unit test for the CDK stack
• Patterns & choices:
• IaC with AWS CDK (constructs, context parameters)
• Serverless compute (Lambda) running on a schedule
• Managed RDS Postgres as a data store
• QuickSight as the BI/visualization layer
README for architectural clues
• Core workflow:
1. Extract Spark job usage from EMR via YARN RM API
2. Extract daily cluster cost via AWS Cost Explorer
3. Extract EC2 usage via EMR/EC2 APIs
4. Store all metrics & costs in RDS tables (DDL provided)
5. (Optional) Purge old data as needed
• Deployment instructions:
• Stage 1: Manually create RDS tables using provided SQL
• Stage 2: Configure environment parameters in cdk.context.json
• Stage 3: Deploy with AWS CDK (pip install → cdk deploy --all)