# surya-kangeyan/serverless-cloud-infrastructure-deployment-with-aws-cloud-formation architecture

> Architecture diagram of the GitHub repository surya-kangeyan/serverless-cloud-infrastructure-deployment-with-aws-cloud-formation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/surya-kangeyan/serverless-cloud-infrastructure-deployment-with-aws-cloud-formation
- Repository: https://github.com/surya-kangeyan/serverless-cloud-infrastructure-deployment-with-aws-cloud-formation

## Overview

##### Main Components
- CloudFormation Template (template.yaml and packaged-template.yaml): This single source-of-truth defines all AWS resources (e.g., IAM roles, S3 buckets, Lambda functions, SQS queues, and DLQ configurations) and handles resource sequencing.
- Lambda Functions:
- queue-populator (found in queue-populator/index.py and packaged as queue-populator.zip): Responsible for scanning S3 (or other triggers) and populating an SQS queue.
- json-to-csv-converter (found in json-to-csv-converter/index.py and packaged as json-to-csv-converter.zip): Processes .json.gz input files to produce CSV output.
- Shell Scripts:
- setup.sh: Likely responsible for preparing the environment.
- deploy_stack.sh: Script to deploy the CloudFormation stack.
- collate.sh: A script to handle post-processing or collating outputs.
- Output Artifact:
- output.csv: The result of the JSON-to-CSV conversion.
- Supporting Files:
- zip archives for the Lambda functions suggest a build/deployment packaging process.
- Other files like queue-populator/ and json-to-csv-converter/ directories show source code organization.

##### Relationships and Interactions
- The CloudFormation template orchestrates the entire resource provisioning. It defines IAM roles, sets up S3 buckets (for input files and Lambda code), configures Lambda functions, and establishes SQS queues (with potential DLQ configured) to handle asynchronous processing.
- The queue-populator Lambda function is triggered (possibly via events in S3) and it scans for incoming .json.gz files, populating the SQS queue.
- The json-to-csv-converter Lambda function reads messages from SQS, processes the files (performing file download from S3, conversion, etc.), and writes the output to output.csv or another designated storage.
- The deployment scripts (setup.sh, deploy_stack.sh, and collate.sh) manage the environment setup, deployment process, code uploads to S3, and final collation of outputs.
- The system leverages AWS services and CloudWatch for logging and debugging, as described in the README.

##### Architectural Patterns and Key Technologies
- Serverless Architecture: Utilizing AWS Lambda for on-demand processing, SQS for queuing, and CloudFormation for infrastructure provisioning.
- Infrastructure as Code: All the AWS resource definitions and dependencies are managed declaratively within CloudFormation templates ensuring consistent and repeatable environments.
- Deployment Best Practices: The deployment flow involves building Lambda packages as zip files, uploading them to S3, and updating CloudFormation stacks, with rollback capabilities in case of errors.
- Debugging and Monitoring: CloudWatch Logs and stack events are essential for tracing deployment and runtime issues.

## Diagram (Mermaid)

```mermaid
flowchart TD
    DS["Deployment Scripts"]:::deployment
    CF["CloudFormation Template"]:::infrastructure
    S3["S3 Bucket (Code/Input/Output)"]:::storage
    SQS["SQS Queue"]:::messaging
    subgraph "AWS Lambda Functions"
        LP1["Lambda: queue-populator"]:::compute
        LP2["Lambda: json-to-csv-converter"]:::compute
    end
    Out["Output Artifact (CSV)"]:::storage
    CW["CloudWatch Logs"]:::monitoring

    %% Deployment Pipeline triggers CloudFormation provisioning
    DS -->|"triggers"| CF

    %% CloudFormation provisions and orchestrates AWS resources
    CF -->|"provisions"| S3
    CF -->|"provisions"| SQS
    CF -->|"deploys"| LP1
    CF -->|"deploys"| LP2
    CF -->|"logs to"| CW

    %% S3 bucket provides input to the queue-populator Lambda
    S3 -->|"input trigger"| LP1

    %% Lambda queue-populator scans and populates the SQS queue
    LP1 -->|"populates"| SQS
    LP1 -->|"logs to"| CW

    %% SQS triggers the json-to-csv-converter Lambda
    SQS -->|"triggers"| LP2
    LP2 -->|"logs to"| CW

    %% Lambda json-to-csv-converter produces the output CSV file stored in S3
    LP2 -->|"produces"| Out

    %% Click Events for Component Mappings
    click CF "https://github.com/surya-kangeyan/serverless-cloud-infrastructure-deployment-with-aws-cloud-formation/blob/main//template.yaml"
    click CF "https://github.com/surya-kangeyan/serverless-cloud-infrastructure-deployment-with-aws-cloud-formation/blob/main//packaged-template.yaml"
    click LP1 "https://github.com/surya-kangeyan/serverless-cloud-infrastructure-deployment-with-aws-cloud-formation/blob/main//queue-populator/index.py"
    click LP1 "https://github.com/surya-kangeyan/serverless-cloud-infrastructure-deployment-with-aws-cloud-formation/blob/main//queue-populator.zip"
    click LP2 "https://github.com/surya-kangeyan/serverless-cloud-infrastructure-deployment-with-aws-cloud-formation/blob/main//json-to-csv-converter/index.py"
    click LP2 "https://github.com/surya-kangeyan/serverless-cloud-infrastructure-deployment-with-aws-cloud-formation/blob/main//json-to-csv-converter.zip"
    click DS "https://github.com/surya-kangeyan/serverless-cloud-infrastructure-deployment-with-aws-cloud-formation/blob/main//setup.sh"
    click DS "https://github.com/surya-kangeyan/serverless-cloud-infrastructure-deployment-with-aws-cloud-formation/blob/main//deploy_stack.sh"
    click DS "https://github.com/surya-kangeyan/serverless-cloud-infrastructure-deployment-with-aws-cloud-formation/blob/main//collate.sh"
    click Out "https://github.com/surya-kangeyan/serverless-cloud-infrastructure-deployment-with-aws-cloud-formation/blob/main//output.csv"

    %% Styles
    classDef infrastructure fill:#f9e79f,stroke:#d35400,stroke-width:2px;
    classDef compute fill:#aed6f1,stroke:#3498db,stroke-width:2px;
    classDef messaging fill:#a9dfbf,stroke:#27ae60,stroke-width:2px;
    classDef storage fill:#fcf3cf,stroke:#f1c40f,stroke-width:2px;
    classDef deployment fill:#ecf0f1,stroke:#7f8c8d,stroke-width:2px;
    classDef monitoring fill:#f5c6cb,stroke:#e74c3c,stroke-width:2px;
```
