How surya-kangeyan/serverless-cloud-infrastructure-deployment-with-aws-cloud-formation works
Main Components
• CloudFormation Template (template.yaml and packaged-template.yaml): This single source-of-truth defines all AWS resources (e.g., IAM roles, S3 buckets, Lambda functions, SQS queues, and DLQ configurations) and handles resource sequencing.
• Lambda Functions:
• queue-populator (found in queue-populator/index.py and packaged as queue-populator.zip): Responsible for scanning S3 (or other triggers) and populating an SQS queue.
• json-to-csv-converter (found in json-to-csv-converter/index.py and packaged as json-to-csv-converter.zip): Processes .json.gz input files to produce CSV output.
• Shell Scripts:
• setup.sh: Likely responsible for preparing the environment.
• deploy_stack.sh: Script to deploy the CloudFormation stack.
• collate.sh: A script to handle post-processing or collating outputs.
• Output Artifact:
• output.csv: The result of the JSON-to-CSV conversion.
• Supporting Files:
• zip archives for the Lambda functions suggest a build/deployment packaging process.
• Other files like queue-populator/ and json-to-csv-converter/ directories show source code organization.
Relationships and Interactions
• The CloudFormation template orchestrates the entire resource provisioning. It defines IAM roles, sets up S3 buckets (for input files and Lambda code), configures Lambda functions, and establishes SQS queues (with potential DLQ configured) to handle asynchronous processing.
• The queue-populator Lambda function is triggered (possibly via events in S3) and it scans for incoming .json.gz files, populating the SQS queue.
• The json-to-csv-converter Lambda function reads messages from SQS, processes the files (performing file download from S3, conversion, etc.), and writes the output to output.csv or another designated storage.
• The deployment scripts (setup.sh, deploy_stack.sh, and collate.sh) manage the environment setup, deployment process, code uploads to S3, and final collation of outputs.
• The system leverages AWS services and CloudWatch for logging and debugging, as described in the README.
Architectural Patterns and Key Technologies
• Serverless Architecture: Utilizing AWS Lambda for on-demand processing, SQS for queuing, and CloudFormation for infrastructure provisioning.
• Infrastructure as Code: All the AWS resource definitions and dependencies are managed declaratively within CloudFormation templates ensuring consistent and repeatable environments.
• Deployment Best Practices: The deployment flow involves building Lambda packages as zip files, uploading them to S3, and updating CloudFormation stacks, with rollback capabilities in case of errors.
• Debugging and Monitoring: CloudWatch Logs and stack events are essential for tracing deployment and runtime issues.