How chime/terraform-aws-alternat works
Main Components
• Terraform Module: At the core, the project is a Terraform module that sets up AWS resources.
• NAT Instance Auto Scaling Groups (ASGs): One per public subnet (typically one per availability zone). Each ASG deploys a single NAT instance with a configurable maximum instance lifetime that triggers regular replacement for patching.
• Standby NAT Gateways: Serve as failover targets. They are activated during NAT instance replacement events.
• replace-route Lambda Function: This Lambda (located in the functions/replace-route directory) listens for lifecycle events (via SNS notifications) from the ASGs and periodically runs health checks (triggered by CloudWatch Events). It updates the private subnet route tables to switch between NAT instances and the standby NAT Gateway.
• SNS Topics & CloudWatch Events: SNS is used to propagate termination notifications via lifecycle hooks (“Terminating:Wait”). Additionally, a CloudWatch Event rule invokes the Lambda function once per minute for connectivity checks.
• Route Tables: Represent the private subnet routing configuration that is dynamically updated by the Lambda based on instance health or termination events.
• User Data Script: The alternat.sh.tftpl script (within scripts and referenced in the module) is used to initialize the NAT instance when it boots, reclaiming the Elastic IP and setting up necessary configurations.
• Auxiliary Artifacts: Dockerfile, SAM testing setup, examples (Terraform files), and additional scripts that support testing and deployment operations.
Relationships and Interactions
• The Terraform module provisions the ASGs that launch NAT instances and creates standby NAT Gateways.
• When a NAT instance reaches its maximum lifetime (or fails), the ASG terminates it, triggering a “Terminating:Wait” lifecycle hook and sending an event through SNS.
• The SNS event triggers the replace-route Lambda function, which updates the appropriate private subnet’s route table to temporarily direct traffic through the standby NAT Gateway.
• The Lambda also runs periodic connectivity tests (via CloudWatch Events) to ensure that the NAT instance has a valid Internet connection (testing URLs like https://www.example.com and https://www.google.com); if both tests fail, it again switches the traffic to the standby NAT Gateway.
• When a newly launched NAT instance boots, its user data script automatically reclaims the Elastic IP and reconfigures the route table to direct traffic through itself.
Architectural Patterns & Design Principles
• Automated Failover: The design uses SNS-triggered Lambda functions and CloudWatch Events to handle failover automatically between NAT instances and standby NAT Gateways.
• Infrastructure as Code: The entire architecture is defined in Terraform (with ancillary testing using Terratest and SAM).
• Loose Coupling: The Lambda function communicates with the ASGs and route tables via well-defined AWS APIs, reducing tight coupling between components.
• Scalability & Resilience: Each availability zone gets its own ASG and associated NAT instance, ensuring high availability while maintaining cost benefits.
Final Recommendations for the Principal Software Engineer
• Automated routing and failover mechanism: how the Lambda handles both lifecycle hook triggers (via SNS) and routine connectivity checks.