How malawson/rabbitmq-ecs-autoclustering works
Project type and purpose
• This is a containerized, infrastructure‐automation tool that runs RabbitMQ in Amazon ECS EC2 Container Service and automatically forms and heals a RabbitMQ cluster based on AWS Auto Scaling Group (ASG) membership.
• It is not a web or mobile app; rather, it’s an operational component that you build into your ECS deployments to manage RabbitMQ clustering.
File structure
• Top-level:
• README.md – explains the project purpose, usage, and AWS prerequisites.
• docker-image/ – directory holding all Docker image build artifacts.
• docker-image/ contains:
• Dockerfile – builds a Docker image atop the official RabbitMQ image, copying in scripts & configuration.
• docker-entrypoint.sh – custom entrypoint that invokes the auto-clustering logic before starting RabbitMQ.
• erlang.cookie – default Erlang cookie for clustering authentication.
• rabbitmq_asg_autocluster.py – Python script that queries the AWS ASG and EC2 APIs to discover peer nodes, issue RabbitMQ cluster-join commands, and recover from failures.
README for additional insights
• Key AWS services & resources:
• ECS cluster (EC2 launch type) backed by an ASG
• IAM roles/policies: DescribeAutoScalingGroups, DescribeInstances
• ECR registry for storing the custom Docker image
• CloudFormation TaskDefinition & Service definitions (host networking mode)
• CloudWatch Logs for RabbitMQ container logs
• Operational flow: On container start (entrypoint), if AWS_ASG_AUTOCLUSTER=true, run Python script to locate peers, cluster or join existing cluster, then hand off control to rabbitmq-server.
a. Main components to include
• AWS::ECR – holds the custom RabbitMQ image
• AWS::IAM Role/InstanceProfile – grants EC2 tasks permissions to query ASG/EC2
• AWS::AutoScalingGroup – group of EC2 instances running the ECS agent
• AWS::ECS Cluster – logical cluster scheduling tasks on ASG instances
• ECS Task Definition & Service – defines the containers, host networking, volumes
• Container (RabbitMQ + entrypoint) – Docker container with rabbitmq_server and auto-cluster script
• Python auto-clustering script – runs inside the container to join/repair the cluster
• RabbitMQ nodes – processes on each container forming the cluster
• AWS::CloudWatch Logs – collects container logs
• AWS::VPC Subnets & Security Groups – network isolation (optional detail)
b. Relationships and interactions
• The ASG ensures a desired count of EC2 instances.
• ECS schedules one RabbitMQ container per EC2 instance (distinctInstance & host networking).
• Container startup → docker-entrypoint.sh → rabbitmq_asg_autocluster.py → AWS ASG/EC2 API calls → cluster join operations via RabbitMQ CLI.
• RabbitMQ nodes communicate peer-to-peer over TCP ports.
• Logs are shipped to CloudWatch Logs.
• ECR → ECS pulls the custom image.
c. Architectural patterns & design principles
• Immutable Docker image (infrastructure as code)
• Host networking for container hostname inheritance (DNS-based node discovery)
• Self-healing cluster via ASG & script-driven peer discovery
• Sidecar-style clustering logic (entrypoint + script) co-located with RabbitMQ process
d. Key technologies & frameworks
• Docker, Dockerfile, host networking
• Python (boto3 or AWS SDK) in rabbitmq_asg_autocluster.py
• RabbitMQ official Debian image (3.6)
• AWS ECS, Auto Scaling Group, IAM, ECR, CloudWatch Logs
• AWS CloudFormation TaskDefinition & Service resources