How aws/aws-parallelcluster works
Project type and purpose
• AWS ParallelCluster is an open-source, AWS-supported HPC cluster management tool.
• It provides:
• A Python CLI (cli/) for user commands (pcluster)
• A Python serverless API (Flask + AWS Lambda controllers)
• A CloudFormation-based orchestration layer (cloudformation/)
• A client SDK generator (api/ using Smithy/OpenAPI)
• Auxiliary CLI for AWS Batch (awsbatch-cli/)
• Utility scripts (util/, pc_support/)
• Integration tests (tests/integration-tests/)
File structure
• Top‐level modules:
• cli/ – core Python CLI and API implementation
• api/ – Smithy/OpenAPI specs + codegen for service client
• awsbatch-cli/ – standalone AWS Batch CLI tool
• cloudformation/ – CFN templates and validation scripts
• util/, pc_support/ – versioning & support scripts
• tests/ – unit & integration tests
• Patterns:
• MVC-style API controllers in cli/src/pcluster/api/controllers
• Template-based CFN stacks in cli/src/pcluster/templates and cloudformation/
• Spec→code generation in api/spec/smithy & api/spec/openapi
• Serverless deployment via AWS Lambda (cli/src/pcluster/api/awslambda)
• Build & CI:
• Python packaging (setup.py, tox.ini)
• Gradle for API codegen (api/build.gradle)
• GitHub workflows in .github/workflows
Extract README insights
• Project is CLI-first but embeds an HTTP API (Flask + Lambda) for programmatic use.
• Uses AWS CDK & CloudFormation under the hood.
• Depends on AWS SDK calls to provision AWS resources (EC2, EFS, FSx, Batch, DynamoDB, S3, CloudWatch).
Main components & interactions
a. Components:
1. User Interface
• Local Python CLI (pcluster commands)
• AWS Batch CLI (awsbatch-cli)
2. API Layer
• Flask app
• AWS Lambda handlers
• OpenAPI/Smithy spec & generated client (api/client)
3. Orchestration Layer
• CloudFormation stacks (head node, compute fleet, image builder, networking, etc.)
• AWS CDK artifacts
4. AWS Services
• EC2, Auto Scaling, EBS, EFS, FSx, S3, DynamoDB, IAM, CloudWatch, AWS Batch, KMS, SSM, Route 53
5. Persistence & State
• DynamoDB for metadata (e.g., cluster state)
• S3 for shared storage & logs
6. Utilities & Support
• Version management, AMI generation, config conversion
b. Relationships:
• CLI invokes API controllers (locally or via REST to Lambda).
• API layer calls AWS SDK modules (cli/src/pcluster/aws/*) to orchestrate resources.
• AWS SDK modules trigger CloudFormation via templated stacks.
• CloudFormation provisions AWS services.
• AWS services report status back via SDK; logs stored in CloudWatch/S3.
• Integration tests simulate CLI→API→AWS interactions.
• API spec source (Smithy/OpenAPI) feeds codegen for client libraries.
c. Architectural patterns & principles:
• Layered architecture (CLI → API → Orchestration → AWS infra)
• Serverless pattern for API (AWS Lambda + API Gateway implied)
• Infrastructure-as-Code (CloudFormation, CDK)
• Code generation for SDK (Smithy → Python client code)
• Modular packaging (separate Gradle module, Python packages)
d. Key technologies & frameworks:
• Python 3.7+ (Click or argparse for CLI), Flask, AWS Lambda
• AWS SDK for Python (boto3)
• AWS CloudFormation and AWS CDK
• Smithy and OpenAPI for spec and codegen (Gradle, OpenAPI generator)
• Docker for CloudFormation tests in api/tests & cloudformation/tests
• CI via GitHub Actions and CodeQL