# aws-samples/esg-compliance-report-processor architecture

> Architecture diagram of the GitHub repository aws-samples/esg-compliance-report-processor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aws-samples/esg-compliance-report-processor
- Repository: https://github.com/aws-samples/esg-compliance-report-processor

## Overview

##### Project Type and Purpose
- This is a full-stack, serverless AWS application with an ECS-hosted Streamlit frontend.
- It ingests compliance reports, orchestrates processing via Step Functions and Lambdas, stores data in DynamoDB/S3, and provides a secured UI behind Cognito.

##### File Structure
- Top-level “cdk/” directory: all CDK stacks and Lambda function definitions, layer configs, tests.
- “streamlit/” directory: the Python Streamlit web app, Dockerfile, setup scripts, and Cognito + ECS configuration.
- “assets/architecture” holds the reference SVG. “examples/” contains sample inputs.

##### Main Components to Model
a. S3 Report Upload Bucket
b. EventBridge Rule & Step Functions State Machine
c. AWS Lambda Functions (twelve-plus small, single-purpose Lambdas)
d. Lambda Layers for third-party libs (PyMuPDF, Textract clients, LangChain, etc.)
e. Amazon Textract & Amazon Bedrock (external AI/ML services)
f. Amazon DynamoDB Tables (supplier_table, compliance_grading_table)
g. Amazon SNS Topic (optional notifications)
h. API Gateway (approve/reject endpoints)
i. Amazon Cognito User Pool & Identity Pool (OIDC integration)
j. VPC, ECS Fargate Cluster & Task Definition (Streamlit UI)
k. Application Load Balancer with ACM certificate
l. Route 53 alias record (DNS entry)

##### Relationships & Data Flow
- User uploads report via Streamlit UI → S3 “report_bucket.”
- S3 “PutObject” event → EventBridge → Step Functions.
- Step Functions invokes Lambdas in sequence:
- report_split → extract_nc → bedrock_supplier_extraction → supplier_details → get_nc → get_status → generate_email → email_approved/email_rejected → send_emails → upload_grading → validate_unrated_issues.
- Lambdas read/write DynamoDB tables and may publish to SNS.
- Textract & Bedrock invoked within specific Lambdas.
- Frontend polls API Gateway endpoints (secured by Cognito) for status and email summary.
- UI served by ECS Fargate behind ALB. ALB enforces HTTPS with ACM cert.
- Cognito sits between ALB and API Gateway to protect endpoints.

##### Architectural Patterns & Principles
- Event-driven, serverless pipeline (S3 → EventBridge → Step Functions → Lambdas).
- Single-responsibility Lambdas (micro-functions).
- Layer reuse for heavy dependencies.
- IAM-backed security and least-privilege roles.
- Decoupling of components via queues/events (EventBridge, SNS).
- Frontend/backend separation (ECS UI vs. serverless backend).

##### Technologies & Frameworks to Annotate
- AWS CDK (Python)
- AWS Lambda & Lambda Layers
- AWS Step Functions
- Amazon S3, DynamoDB, SNS, API Gateway
- Amazon Textract & Amazon Bedrock
- Amazon Cognito (OIDC)
- ECS Fargate, ALB, ACM, Route 53
- Streamlit, PyMuPDF, LangChain, Pandas

##### Must-Include Elements
- Callouts for configuration: ACM cert, Secrets Manager for OIDC credentials.

## Diagram (Mermaid)

```mermaid
graph TB
    %% User Layer
    subgraph "User Layer"
        direction TB
        User["End User"]:::user
    end

    %% UI Layer (VPC)
    subgraph "UI Layer (VPC)" 
        direction TB
        Cognito["Amazon Cognito\n(OIDC Auth)"]:::api
        ALB["Application Load Balancer\n(HTTPS, ACM)"]:::network
        ECS["ECS Fargate\nStreamlit App"]:::ui
    end

    %% API Layer
    subgraph "API Layer"
        direction TB
        APIGW["API Gateway\nApprove/Reject"]:::api
    end

    %% Processing Layer
    subgraph "Processing Layer"
        direction TB
        EB["EventBridge Rule"]:::compute
        SF["Step Functions\nState Machine"]:::compute
        subgraph "Lambda Functions" 
            direction TB
            RS["report_split"]:::compute
            EX["extract_nc"]:::compute
            BS["bedrock_supplier_extraction"]:::compute
            SD["supplier_details"]:::compute
            GN["get_nc"]:::compute
            GS["get_status"]:::compute
            GE["generate_email"]:::compute
            EA["email_approved"]:::compute
            ER["email_rejected"]:::compute
            SE["send_emails"]:::compute
            UG["upload_grading"]:::compute
            VU["validate_unrated_issues"]:::compute
        end
        subgraph "Lambda Layers"
            direction TB
            B3["boto3"]:::layer
            FA["faiss"]:::layer
            FZ["fuzzywuzzy"]:::layer
            LC["langchain"]:::layer
            PA["pandas"]:::layer
            PH["pyhtml"]:::layer
            MP["pymupdf"]:::layer
            PY["pyyaml"]:::layer
            TXL["textractor"]:::layer
        end
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        S3["S3 Report Upload Bucket"]:::data
        DDB1["DynamoDB\nsupplier_table"]:::data
        DDB2["DynamoDB\ncompliance_grading_table"]:::data
        SNS["SNS Topic"]:::data
    end

    %% External Services
    subgraph "External AI Services"
        direction TB
        TEX["Amazon Textract"]:::external
        BED["Amazon Bedrock"]:::external
    end

    %% DNS
    Route53["Route 53 Alias"]:::network

    %% Connections
    User -->|"Login"| Cognito
    Cognito --> ALB
    ALB --> ECS
    ECS -->|"PUT Object"| S3
    ECS -->|"Poll Status"| APIGW

    APIGW -->|"Invoke"| GS
    APIGW -->|"Invoke"| GE

    S3 -->|"S3 PutObject →"| EB
    EB -->|"Start Execution"| SF

    SF -->|"Step: report_split"| RS
    RS -->|"Step: extract_nc"| EX
    EX -->|"Step: bedrock_supplier_extraction"| BS
    BS -->|"Step: supplier_details"| SD
    SD -->|"Step: get_nc"| GN
    GN -->|"Step: get_status"| GS
    GS -->|"Step: generate_email"| GE
    GE -->|"Step: email_approved/email_rejected"| EA
    GE -->|"Step: email_approved/email_rejected"| ER
    EA -->|"Step: send_emails"| SE
    ER -->|"Step: send_emails"| SE
    SE -->|"Step: upload_grading"| UG
    UG -->|"Step: validate_unrated_issues"| VU

    %% Lambda interactions
    RS -->|Reads/Writes| DDB1
    EX -->|Reads| DDB1
    BS -->|Writes| DDB1
    SD -->|Reads| DDB1
    GN -->|Reads| DDB2
    UG -->|Writes| DDB2
    any -->|Publish| SNS

    BS -->|"Invoke"| BED
    EX -->|"Invoke"| TEX

    %% Route53
    Route53 --> ALB

    %% Click Events
    click S3 "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/esg_compliance_cdk/reportupload/reportupload.py"
    click EB "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/esg_compliance_cdk/reportupload/reportupload.py"
    click RS "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/lambdas/report_split/lambda_function.py"
    click EX "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/lambdas/extract_nc/lambda_function.py"
    click BS "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/lambdas/bedrock_supplier_extraction/lambda_function.py"
    click SD "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/lambdas/supplier_details/lambda_function.py"
    click GN "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/lambdas/get_nc/lambda_function.py"
    click GS "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/lambdas/get_status/lambda_function.py"
    click GE "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/lambdas/generate_email/lambda_function.py"
    click EA "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/lambdas/email_approved/lambda_function.py"
    click ER "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/lambdas/email_rejected/lambda_function.py"
    click SE "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/lambdas/send_emails/lambda_function.py"
    click UG "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/lambdas/upload_grading/lambda_function.py"
    click VU "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/lambdas/validate_unrated_issues/lambda_function.py"
    click B3 "https://github.com/aws-samples/esg-compliance-report-processor/tree/main/cdk/esg-compliance-cdk/lambda_layers/boto3"
    click FA "https://github.com/aws-samples/esg-compliance-report-processor/tree/main/cdk/esg-compliance-cdk/lambda_layers/faiss"
    click FZ "https://github.com/aws-samples/esg-compliance-report-processor/tree/main/cdk/esg-compliance-cdk/lambda_layers/fuzzywuzzy"
    click LC "https://github.com/aws-samples/esg-compliance-report-processor/tree/main/cdk/esg-compliance-cdk/lambda_layers/langchain"
    click PA "https://github.com/aws-samples/esg-compliance-report-processor/tree/main/cdk/esg-compliance-cdk/lambda_layers/pandas"
    click PH "https://github.com/aws-samples/esg-compliance-report-processor/tree/main/cdk/esg-compliance-cdk/lambda_layers/pyhtml"
    click MP "https://github.com/aws-samples/esg-compliance-report-processor/tree/main/cdk/esg-compliance-cdk/lambda_layers/pymupdf"
    click PY "https://github.com/aws-samples/esg-compliance-report-processor/tree/main/cdk/esg-compliance-cdk/lambda_layers/pyyaml"
    click TXL "https://github.com/aws-samples/esg-compliance-report-processor/tree/main/cdk/esg-compliance-cdk/lambda_layers/textractor"
    click BED "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/lambdas/bedrock_supplier_extraction/modules/bedrock_extraction.py"
    click DDB1 "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/config/tables.yaml"
    click DDB2 "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/config/tables.yaml"
    click APIGW "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/esg_compliance_cdk/esg_compliance_cdk_stack.py"
    click Cognito "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/esg_compliance_cdk/cognito/cognito.py"
    click ALB "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/esg_compliance_cdk/network/network.py"
    click ECS "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/esg_compliance_cdk/network/network.py"
    click Route53 "https://github.com/aws-samples/esg-compliance-report-processor/blob/main/cdk/esg-compliance-cdk/esg_compliance_cdk/network/network.py"

    %% Styles
    classDef ui fill:#D0E6FF,stroke:#0366d6,color:#0366d6;
    classDef api fill:#DFF5D4,stroke:#228B22,color:#228B22;
    classDef compute fill:#FFE5CC,stroke:#FF8C00,color:#FF8C00;
    classDef data fill:#E9D5FF,stroke:#800080,color:#800080;
    classDef external fill:#F0F0F0,stroke:#A9A9A9,color:#696969;
    classDef network fill:#E6F7FF,stroke:#1E90FF,color:#1E90FF;
    classDef layer fill:#FFF2CC,stroke:#DAA520,color:#DAA520;
    classDef user fill:#FFFFCC,stroke:#FFD700,color:#FFD700;
```
