# aws-samples/amazon-kinesis-load-testing-with-locust architecture

> Architecture diagram of the GitHub repository aws-samples/amazon-kinesis-load-testing-with-locust, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aws-samples/amazon-kinesis-load-testing-with-locust
- Repository: https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust

## Overview

##### Project type and purpose
- This is an AWS-centric load-testing solution that uses AWS CDK (TypeScript) to provision infrastructure and Locust (Python) to generate load against an Amazon Kinesis Data Stream.
- The core use case is “simulate high throughput event streams to Kinesis” either locally or in the cloud, leveraging CDK for infra and Locust for load.

##### File structure
- Top-level:
- .github, CI/CD workflows and templates
- package.json, tsconfig.json, cdk.json → TypeScript/CDK project
- infrastructure/ → CDK stack definition (infrastructure/kinesis-load-testing-with-locust.ts)
- load-test/ → Python locust files, environment vars, helper scripts
- test/ → Jest tests for the CDK code
- Patterns:
- IaC folder (infrastructure) separate from application code (load-test)
- CDK unit tests under test/
- Shell scripts to start/stop distributed Locust workers

##### README for additional insights
- Contains an architecture overview: CDK deploys a VPC, EC2 instance (Locust master + workers as systemd service), and a Kinesis Data Stream.
- Users connect via SSM Session Manager to adjust Locust settings.
- Locust uses Boto3 under the hood to do PutRecords calls in batches.

a. Main components
- Developer machine / GitHub repo (source code + CI)
- AWS CDK pipeline (GitHub Actions)
- AWS Environment:
– VPC (default)
– EC2 Locust Master (systemd service launching master + N secondaries)
– Locust Dashboard (HTTP on 8089, secured)
– Kinesis Data Stream (DemoStream)
– IAM roles, Security Groups, SSM Session Manager
- External services:
– AWS Kinesis
– AWS SSM
b. Relationships and interactions
- GitHub Actions → triggers “cdk deploy” → AWS CDK creates infra
- EC2 Instance runs Locust master and distributes work to secondaries
- Locust processes → batch PutRecords → Kinesis Data Stream
- Operator → SSH via SSM → modify locust.env → systemctl restart Locust
- Locust Dashboard ← HTTP ← Operator
c. Architectural patterns and design principles
- Infrastructure as Code (AWS CDK)
- Distributed load generation (Locust master/worker model)
- Boto3 wrapper for API calls to Kinesis
- Separation of concerns: CDK infra vs load-test logic
d. Key technologies & libraries
- AWS CDK (TypeScript)
- Python Locust framework + Boto3
- EC2 (C7G instance family)
- Jest for CDK unit tests
- GitHub Actions for CI/CD

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Developer Environment"
        Dev["Developer"]:::dev
        LocalTest["locust-load-test.py"]:::compute
        Dev -->|"runs locally"| LocalTest
        LocalTest -->|"PutRecords (Boto3 API)"| Kinesis
    end

    subgraph "CI/CD Pipeline"
        Repo["GitHub Repo"]:::tooling
        CI["GitHub Actions"]:::tooling
        JestTests["Jest Unit Tests"]:::tooling
        Snapshot["Jest Snapshot Tests"]:::tooling
        Repo -->|push| CI
        CI -->|"build & test"| JestTests
        CI -->|"build & test"| Snapshot
        CI -->|"cdk deploy"| CDKDeploy
    end

    subgraph "AWS Cloud"
        subgraph "VPC (default)"
            CDKDeploy["AWS CDK Stack"]:::tooling
            VPC["VPC (default)"]:::aws
            EC2["EC2 Locust Master (C7g)"]:::compute
            SG["Security Group\n(8089 open)"]:::aws
            ServiceFile["locust.service"]:::tooling
            subgraph "Locust Processes"
                Master["Locust Master"]:::compute
                Workers["Locust Workers\n(batch size, count)"]:::compute
                LocustScript["locust-load-test.py"]:::compute
                Env["locust.env"]:::tooling
                Req["requirements.txt"]:::tooling
                Start["start-locust.sh"]:::tooling
                Stop["stop-locust.sh"]:::tooling
                Master --> Workers
                Workers -->|"PutRecords API"| Kinesis
                Master -->|"launch via systemd"| ServiceFile
                ServiceFile -.-> LocustScript
                LocustScript --> Env
                LocustScript --> Req
                LocustScript --> Start
                LocustScript --> Stop
            end
            Dashboard["Locust Dashboard (8089)"]:::compute
            Kinesis["Kinesis Data Stream\nDemoStream"]:::aws
            SSM["AWS SSM"]:::aws
            Operator["Operator"]:::dev

            CDKDeploy --> VPC
            VPC --> EC2
            EC2 --> SG
            SG --> Dashboard
            EC2 --> Dashboard
            EC2 --> ServiceFile
            Dashboard -->|"HTTP 8089"| Operator
            Operator -->|"SSH via SSM"| EC2
            Operator -->|"HTTPS"| Dashboard
            EC2 --> Master
            Kinesis
        end
    end

    %% Click Events
    click CDKDeploy "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/infrastructure/kinesis-load-testing-with-locust.ts"
    click ServiceFile "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/infrastructure/locust.service"
    click LocustScript "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/load-test/locust-load-test.py"
    click Env "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/load-test/locust.env"
    click Req "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/load-test/requirements.txt"
    click Start "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/load-test/start-locust.sh"
    click Stop "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/load-test/stop-locust.sh"
    click JestTests "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/test/kinesis-load-testing-with-locust.test.ts"
    click Snapshot "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/test/__snapshots__/kinesis-load-testing-with-locust.test.ts.snap"
    click CI "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/.github/workflows/build-and-test-cdk.yml"
    click Repo "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/README.md"
    click CdkJson "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/cdk.json"
    click PkgJson "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/package.json"
    click Plock "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/package-lock.json"
    click Tsconfig "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/tsconfig.json"
    click JestConfig "https://github.com/aws-samples/amazon-kinesis-load-testing-with-locust/blob/main/jest.config.js"

    classDef aws fill:#ADD8E6,stroke:#000;
    classDef compute fill:#90EE90,stroke:#000;
    classDef tooling fill:#D3D3D3,stroke:#000;
    classDef dev fill:#FFA500,stroke:#000;
```
