# aws/aws-parallelcluster architecture

> Architecture diagram of the GitHub repository aws/aws-parallelcluster, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aws/aws-parallelcluster
- Repository: https://github.com/aws/aws-parallelcluster

## Overview

##### Project type and purpose
- AWS ParallelCluster is an open-source, AWS-supported HPC cluster management tool.
- It provides:
- A Python CLI (`cli/`) for user commands (`pcluster`)
- A Python serverless API (Flask + AWS Lambda controllers)
- A CloudFormation-based orchestration layer (`cloudformation/`)
- A client SDK generator (`api/` using Smithy/OpenAPI)
- Auxiliary CLI for AWS Batch (`awsbatch-cli/`)
- Utility scripts (`util/`, `pc_support/`)
- Integration tests (`tests/integration-tests/`)

##### File structure
- **Top‐level modules**:
- `cli/` – core Python CLI and API implementation
- `api/` – Smithy/OpenAPI specs + codegen for service client
- `awsbatch-cli/` – standalone AWS Batch CLI tool
- `cloudformation/` – CFN templates and validation scripts
- `util/`, `pc_support/` – versioning & support scripts
- `tests/` – unit & integration tests
- **Patterns**:
- MVC-style API controllers in `cli/src/pcluster/api/controllers`
- Template-based CFN stacks in `cli/src/pcluster/templates` and `cloudformation/`
- Spec→code generation in `api/spec/smithy` & `api/spec/openapi`
- Serverless deployment via AWS Lambda (`cli/src/pcluster/api/awslambda`)
- **Build & CI**:
- Python packaging (`setup.py`, `tox.ini`)
- Gradle for API codegen (`api/build.gradle`)
- GitHub workflows in `.github/workflows`

##### Extract README insights
- Project is CLI-first but embeds an HTTP API (Flask + Lambda) for programmatic use.
- Uses AWS CDK & CloudFormation under the hood.
- Depends on AWS SDK calls to provision AWS resources (EC2, EFS, FSx, Batch, DynamoDB, S3, CloudWatch).

##### Main components & interactions
a. Components:
1. User Interface
- Local Python CLI (`pcluster` commands)
- AWS Batch CLI (`awsbatch-cli`)
2. API Layer
- Flask app
- AWS Lambda handlers
- OpenAPI/Smithy spec & generated client (`api/client`)
3. Orchestration Layer
- CloudFormation stacks (head node, compute fleet, image builder, networking, etc.)
- AWS CDK artifacts
4. AWS Services
- EC2, Auto Scaling, EBS, EFS, FSx, S3, DynamoDB, IAM, CloudWatch, AWS Batch, KMS, SSM, Route 53
5. Persistence & State
- DynamoDB for metadata (e.g., cluster state)
- S3 for shared storage & logs
6. Utilities & Support
- Version management, AMI generation, config conversion
b. Relationships:
- CLI invokes API controllers (locally or via REST to Lambda).
- API layer calls AWS SDK modules (`cli/src/pcluster/aws/*`) to orchestrate resources.
- AWS SDK modules trigger CloudFormation via templated stacks.
- CloudFormation provisions AWS services.
- AWS services report status back via SDK; logs stored in CloudWatch/S3.
- Integration tests simulate CLI→API→AWS interactions.
- API spec source (Smithy/OpenAPI) feeds codegen for client libraries.
c. Architectural patterns & principles:
- Layered architecture (CLI → API → Orchestration → AWS infra)
- Serverless pattern for API (AWS Lambda + API Gateway implied)
- Infrastructure-as-Code (CloudFormation, CDK)
- Code generation for SDK (Smithy → Python client code)
- Modular packaging (separate Gradle module, Python packages)
d. Key technologies & frameworks:
- Python 3.7+ (Click or argparse for CLI), Flask, AWS Lambda
- AWS SDK for Python (boto3)
- AWS CloudFormation and AWS CDK
- Smithy and OpenAPI for spec and codegen (Gradle, OpenAPI generator)
- Docker for CloudFormation tests in `api/tests` & `cloudformation/tests`
- CI via GitHub Actions and CodeQL

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interfaces
    subgraph "User Interfaces"
        UI_CLI["pcluster CLI"]:::python
        CLI_Commands["CLI Command Modules"]:::python
        UI_Batch["awsbatch-cli"]:::python
    end

    %% API Layer
    subgraph "API Layer"
        FlaskApp["Flask App Entry Point"]:::python
        API_Controllers["API Controllers"]:::python
        LambdaHandler["Lambda Entrypoint"]:::python
        OpenAPI_Local["OpenAPI Spec (local)"]:::codegen
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        SDKWrappers["CLI AWS SDK wrappers"]:::python
        CLI_Templates["Template-based CFN stacks"]:::python
        CFNManager["CloudFormation Stack Manager"]:::python
    end

    %% Infrastructure Specs & Codegen
    subgraph "Infrastructure Specs & Codegen"
        Smithy["Smithy Models"]:::codegen
        OpenAPISrc["OpenAPI Specs"]:::codegen
        Codegen["Client SDK Generator"]:::codegen
        GeneratedSDK["Generated Client SDK"]:::codegen
        APIDeploy["API Deployment Artifacts"]:::codegen
    end

    %% AWS Services
    subgraph "AWS Services"
        EC2["EC2 & Auto Scaling"]:::aws
        Storage["EBS, EFS, FSx, S3"]:::aws
        BatchService["AWS Batch"]:::aws
        Networking["Route53, IAM, KMS"]:::aws
        Monitoring["CloudWatch & SSM"]:::aws
    end

    %% Persistence & State
    subgraph "Persistence & State"
        S3["S3 Bucket"]:::db
        DynamoDB["DynamoDB"]:::db
    end

    %% Utilities & Tests
    subgraph "Utilities & Tests"
        Utils["Utility Scripts"]:::gray
        PcSupport["Version Support Files"]:::gray
        IntegrationTests["Integration Tests"]:::gray
        CFNTests["CFN Unit & Integration Tests"]:::gray
    end

    CI_CD["GitHub Actions & CI"]:::gray

    %% Relationships
    UI_CLI -->|uses| CLI_Commands
    UI_CLI -->|invokes| FlaskApp
    UI_CLI -->|invokes| LambdaHandler
    UI_CLI -->|calls| SDKWrappers
    UI_CLI -->|uses| Utils
    UI_CLI -->|uses| PcSupport

    UI_Batch -->|CLI tool| SDKWrappers

    FlaskApp -->|routes to| API_Controllers
    LambdaHandler -->|handles| API_Controllers
    FlaskApp -->|serves spec| OpenAPI_Local

    API_Controllers -->|calls| SDKWrappers

    SDKWrappers -->|triggers| CLI_Templates
    SDKWrappers -->|invokes| CFNManager

    CLI_Templates -->|defines stacks| CFNManager
    CFNManager -->|provisions| EC2
    CFNManager -->|provisions| Storage
    CFNManager -->|provisions| BatchService
    CFNManager -->|configures| Networking
    CFNManager -->|enables| Monitoring

    EC2 -->|logs & metrics| Monitoring
    Storage -->|stores logs| S3
    BatchService -->|job metadata| DynamoDB

    Smithy -->|feeds| Codegen
    OpenAPISrc -->|feeds| Codegen
    Codegen -->|generates| GeneratedSDK
    APIDeploy -->|deploys| LambdaHandler

    IntegrationTests -->|simulate| UI_CLI
    IntegrationTests -->|test| FlaskApp
    CFNTests -->|test| CFNManager

    CI_CD -->|triggers| Codegen
    CI_CD -->|runs| IntegrationTests
    CI_CD -->|runs| CFNTests

    %% Click Events
    click UI_CLI "https://github.com/aws/aws-parallelcluster/blob/develop/cli/src/pcluster/cli/entrypoint.py"
    click CLI_Commands "https://github.com/aws/aws-parallelcluster/tree/develop/cli/src/pcluster/cli/commands/"
    click SDKWrappers "https://github.com/aws/aws-parallelcluster/tree/develop/cli/src/pcluster/aws/"
    click API_Controllers "https://github.com/aws/aws-parallelcluster/tree/develop/cli/src/pcluster/api/controllers/"
    click FlaskApp "https://github.com/aws/aws-parallelcluster/blob/develop/cli/src/pcluster/api/flask_app.py"
    click LambdaHandler "https://github.com/aws/aws-parallelcluster/blob/develop/cli/src/pcluster/api/awslambda/entrypoint.py"
    click OpenAPI_Local "https://github.com/aws/aws-parallelcluster/blob/develop/cli/src/pcluster/api/openapi/openapi.yaml"
    click Smithy "https://github.com/aws/aws-parallelcluster/tree/develop/api/spec/smithy/"
    click OpenAPISrc "https://github.com/aws/aws-parallelcluster/tree/develop/api/spec/openapi/"
    click GeneratedSDK "https://github.com/aws/aws-parallelcluster/tree/develop/api/client/src/pcluster_client/"
    click APIDeploy "https://github.com/aws/aws-parallelcluster/blob/develop/api/infrastructure/parallelcluster-api.yaml"
    click CFNManager "https://github.com/aws/aws-parallelcluster/tree/develop/cloudformation/"
    click CLI_Templates "https://github.com/aws/aws-parallelcluster/tree/develop/cli/src/pcluster/templates/"
    click UI_Batch "https://github.com/aws/aws-parallelcluster/tree/develop/awsbatch-cli/src/awsbatch/"
    click Utils "https://github.com/aws/aws-parallelcluster/tree/develop/util/"
    click PcSupport "https://github.com/aws/aws-parallelcluster/tree/develop/pc_support/"
    click IntegrationTests "https://github.com/aws/aws-parallelcluster/tree/develop/tests/integration-tests/"
    click CFNTests "https://github.com/aws/aws-parallelcluster/tree/develop/cloudformation/tests/"

    %% Styles
    classDef python fill:#d5f5e3,stroke:#27ae60,stroke-width:1px
    classDef codegen fill:#d6eaf8,stroke:#2e86c1,stroke-width:1px
    classDef aws fill:#f9e79f,stroke:#d68910,stroke-width:1px
    classDef db fill:#fadbd8,stroke:#cb4335,stroke-width:1px
    classDef gray fill:#d5dbdb,stroke:#566573,stroke-width:1px
```
