# aws-samples/attribute-amazon-emr-costs-to-your-end-users architecture

> Architecture diagram of the GitHub repository aws-samples/attribute-amazon-emr-costs-to-your-end-users, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aws-samples/attribute-amazon-emr-costs-to-your-end-users
- Repository: https://github.com/aws-samples/attribute-amazon-emr-costs-to-your-end-users

## Overview

##### Project type and purpose
- This is an AWS‑focused, open‑source cost‑attribution solution rather than a traditional full‑stack web app.
- It uses AWS CDK (in Python) to deploy a serverless extraction pipeline: a Lambda function that pulls metrics and cost data from EMR, EC2 and Cost Explorer, writes to a Postgres RDS data model, and surfaces dashboards in QuickSight.

##### File structure
- Top level:
- app.py – CDK entry point
- cdk.json + cdk.context.json – CDK configuration
- cdk_emrcostblog/ – CDK stack code (cdk_emrcostblog_stack.py)
- Lambda/ – business‑logic code (lambda_function.py)
- emr-cost-rds-tables-ddl.sql + sample-cost-calculation-query.sql – data model and example queries
- requirements*.txt – Python dependencies
- tests/ – unit test for the CDK stack
- Patterns & choices:
- IaC with AWS CDK (constructs, context parameters)
- Serverless compute (Lambda) running on a schedule
- Managed RDS Postgres as a data store
- QuickSight as the BI/visualization layer

##### README for architectural clues
- Core workflow:
1. Extract Spark job usage from EMR via YARN RM API
2. Extract daily cluster cost via AWS Cost Explorer
3. Extract EC2 usage via EMR/EC2 APIs
4. Store all metrics & costs in RDS tables (DDL provided)
5. (Optional) Purge old data as needed
- Deployment instructions:
- Stage 1: Manually create RDS tables using provided SQL
- Stage 2: Configure environment parameters in cdk.context.json
- Stage 3: Deploy with AWS CDK (pip install → cdk deploy --all)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Infrastructure as Code Layer
    subgraph "Infrastructure as Code (AWS CDK)"
        direction TB
        CDK["AWS CDK"]:::iac
        cdkJson["cdk.json"]:::iac
        cdkContext["cdk.context.json"]:::iac
        stack["CDK Stack\n(cdk_emrcostblog_stack.py)"]:::iac
        requirements["requirements.txt"]:::iac
        requirementsDev["requirements-dev.txt"]:::iac
        tests["Unit Tests\n(test_cdk_workshop_stack.py)"]:::iac
    end

    %% VPC Boundary
    subgraph "VPC (Subnets & Security Groups)"
        direction TB
        EMR["EMR Cluster"]:::compute
        Schedule["CloudWatch Event\n(Cron)"]:::compute
        Lambda["Cost-Attribution Lambda\n(lambda_function.py)"]:::compute
    end

    %% Data Layer
    subgraph "Data Store"
        direction TB
        RDS["RDS Postgres\n(emr-cost-rds-tables-ddl.sql)"]:::datastore
    end

    %% Analytics Layer
    subgraph "Analytics"
        direction TB
        QS["QuickSight Dashboard"]:::analytics
    end

    %% External AWS Services
    subgraph "External Services"
        direction TB
        YARN["YARN RM API"]:::external
        CostExplorer["Cost Explorer API"]:::external
        EC2API["EMR & EC2 APIs"]:::external
    end

    %% Relationships
    CDK -->|deploys| EMR
    CDK -->|deploys| Lambda
    CDK -->|deploys| RDS
    CDK -->|configures| QS

    Schedule -->|triggers| Lambda
    Lambda -->|Get usage metrics| YARN
    Lambda -->|Get cost data| CostExplorer
    Lambda -->|Get instance usage| EC2API
    Lambda -->|Writes data| RDS
    QS -->|Queries data| RDS

    %% Click Events
    click CDK "https://github.com/aws-samples/attribute-amazon-emr-costs-to-your-end-users/blob/main/app.py"
    click cdkJson "https://github.com/aws-samples/attribute-amazon-emr-costs-to-your-end-users/blob/main/cdk.json"
    click cdkContext "https://github.com/aws-samples/attribute-amazon-emr-costs-to-your-end-users/blob/main/cdk.context.json"
    click stack "https://github.com/aws-samples/attribute-amazon-emr-costs-to-your-end-users/blob/main/cdk_emrcostblog/cdk_emrcostblog_stack.py"
    click Lambda "https://github.com/aws-samples/attribute-amazon-emr-costs-to-your-end-users/blob/main/Lambda/lambda_function.py"
    click RDS "https://github.com/aws-samples/attribute-amazon-emr-costs-to-your-end-users/blob/main/emr-cost-rds-tables-ddl.sql"
    click QS "https://github.com/aws-samples/attribute-amazon-emr-costs-to-your-end-users/blob/main/sample-cost-calculation-query.sql"
    click requirements "https://github.com/aws-samples/attribute-amazon-emr-costs-to-your-end-users/blob/main/requirements.txt"
    click requirementsDev "https://github.com/aws-samples/attribute-amazon-emr-costs-to-your-end-users/blob/main/requirements-dev.txt"
    click tests "https://github.com/aws-samples/attribute-amazon-emr-costs-to-your-end-users/blob/main/tests/unit/test_cdk_workshop_stack.py"

    %% Styles
    classDef iac fill:#d4f1be,stroke:#339966,stroke-width:2px;
    classDef compute fill:#bae1ff,stroke:#3399ff,stroke-width:2px;
    classDef datastore fill:#ffd8b1,stroke:#ff9933,stroke-width:2px;
    classDef analytics fill:#e0b3ff,stroke:#9933ff,stroke-width:2px;
    classDef external fill:#f0f0f0,stroke:#999999,stroke-width:1px;
```
