How aws/aws-node-termination-handler works
Main Components
• Kubernetes Cluster – This is the runtime environment where the termination handler is deployed. Represent the cluster with nodes (each running Kubernetes, e.g., EC2 instances).
• AWS Node Termination Handler – It has two modes:
• IMDS Processor Mode: Deployed as a DaemonSet on all cluster nodes. It polls EC2 instance metadata endpoints (like /spot, /events, etc.) on every node.
• Queue Processor Mode: Deployed as a Deployment (potentially with multiple replicas) that listens to an SQS queue for termination events via Amazon EventBridge or direct ASG lifecycle hook messages.
• AWS Services:
• EC2 Instance Metadata (IMDS) – This is where each node fetches local instance metadata about scheduled terminations or state changes.
• Amazon Simple Queue Service (SQS) – Used by Queue Processor mode to receive messages about termination events.
• Amazon EventBridge and ASG lifecycle hooks – These services generate and forward notifications (via EventBridge rules, lifecycle hooks) that are eventually posted into SQS.
• Auto Scaling Group (ASG) – Manages EC2 instances and uses lifecycle hooks to delay termination until the node is cordoned/drained.
• Kubernetes API – The termination handler uses the Kubernetes API to cordon (prevent new pods from being scheduled) and drain nodes when an interruption is detected.
• Additional Infrastructure/Integration:
• Helm charts & Config files – As shown in the config/helm folder, which help deploy and configure the application.
• Logging and Observability Components – Exposed Prometheus metrics, as well as integrations with webhook endpoints (e.g., Slack notifications).
Relationships and Data Flow
• For IMDS Processor Mode:
• Each node (represented as a pod running the IMDS handler) polls the EC2 instance metadata endpoints.
• When a termination event (spot interruption, scheduled maintenance, or rebalance recommendation) is detected, the pod issues a call to the Kubernetes API to cordon and drain its own node.
• For Queue Processor Mode:
• An AWS SQS queue collects events from EventBridge or ASG lifecycle hooks.
• The Queue Processor (running as a Deployment) reads messages from SQS.
• Once a termination message is received, it triggers an action via the Kubernetes API to cordon and drain the associated node.
• Additionally, if enabled, the handler sends periodic heartbeats back to ASG (extending termination wait as necessary).
• Additional flows include logging, metric collection (via Prometheus endpoints), and optional webhook notifications.
Architectural Patterns and Technologies
• Separation of Concerns – The design clearly separates the monitoring (via IMDS for each node) and event processing (via Queue Processor) into distinct deployment patterns.
• Kubernetes Native Deployment Patterns – Use of DaemonSets for node-local monitoring and Deployments for centralized event processing.
• Cloud Integration – Use of AWS services (EC2, SQS, EventBridge, ASG) integrated tightly with Kubernetes for life‐cycle management.
• Extensibility – The helm charts and configuration files (in config/helm) demonstrate how the project can be easily configured and extended.
Detailed Elements
• Kubernetes Cluster:
• Nodes (EC2 instances) with “aws-node-termination-handler” running in two possible modes (DaemonSet for IMDS; Deployment for Queue Processing).
• Kubernetes API server, which is the target for node cordoning/draining.
• AWS Components:
• EC2 Instance Metadata Service (illustrate the metadata endpoints being polled by the IMDS mode).
• ASG and Lifecycle Hooks, with notes about termination grace periods and heartbeat extension.
• Integration/Monitoring:
• How Prometheus metrics are collected (via PodMonitor/ServiceMonitor).
• Optionally, show external webhooks for notifications.
• Data/Control Flows: