How dgaliata/the-magnolia-project works
Project Type and Purpose
• The Magnolia Project is a full-stack, cloud-native infrastructure-as-code (IaC) deployment combined with multiple FastAPI-based applications, CI/CD pipelines, security controls, and monitoring.
• Purpose: Automate and secure the deployment of a biomedical company’s web apps (Employee Directory, Scientific Data Management, Authentication Service) on AWS, with DynamoDB/S3 storage, Prometheus/Grafana monitoring, and GitHub Actions pipelines.
File Structure
• Top-level: README.md, Terraform folder, GitHub workflows, .gitignore
• terraform/environments: separate state (dev, production)
• terraform/modules: reusable building blocks for compute, database, load_balancing, monitoring, security, storage, vpc
• terraform/provider.tf: AWS provider configuration
• CI/CD: .github/workflows/trivy_tf_scan.yml triggers security scans on Terraform code
Extract Architectural Insights from the README
• AWS Components: VPC, subnets, EC2, security groups, IAM roles, DynamoDB tables, S3 buckets
• FastAPI Apps: served on EC2 (or containerized), backed by DynamoDB/S3, template rendering via Jinja2
• CI/CD: GitHub Actions for Terraform plan/apply and FastAPI build/deploy, with integrated security checks (Trivy, Terraform security scans)
• Monitoring & Logging: Prometheus (metrics exporter on apps & infra), Grafana dashboards, CloudWatch, CloudTrail, GuardDuty, AWS SNS alerts
Main Components & Their Interactions
a. GitHub Actions Pipeline
• Stages: lint/format → security scans (Trivy, tfsec) → Terraform plan/apply → FastAPI application build & deploy
• Artifacts flow into AWS CodeDeploy (or custom deployments)
b. Terraform IaC
• Orchestrates all AWS resources via modules
• Modules enforce separation of concerns (networking, security, compute, storage, monitoring)
c. AWS Networking & Security
• VPC module: public/private subnets, routing tables, NAT
• Security module: security groups, NACLs, IAM roles/policies
d. Compute & Load Balancing
• Compute module: EC2 Auto Scaling Group or ECS/EKS cluster for FastAPI services & monitoring agents
• Load Balancing module: Application Load Balancer distributing traffic to FastAPI instances
e. Data Storage
• Database module: DynamoDB tables for application data (employees, experiments)
• Storage module: S3 buckets for experiment assets, lifecycle policies to Glacier
f. Monitoring & Alerting
• Monitoring module: EC2 instances or sidecar containers running Prometheus exporters
• Grafana (deployed on EC2 or Grafana Cloud) visualizing metrics from Prometheus and AWS CloudWatch
• Alerting through AWS SNS, GuardDuty events, CloudWatch alarms
Architectural Patterns & Design Principles
• Infrastructure-as-Code: Terraform modules promote DRY, versioned, environment-specific deployments
• Microservices/SOA: Each FastAPI app is logically separated (Employee, Scientific Data, Auth) but co-hosted on shared compute resources
• Security-First: Principle of least privilege via IAM roles, encrypted-at-rest data stores, CI/CD security scans
• Observability: Metrics (Prometheus), logs (CloudWatch, CloudTrail), dashboards (Grafana), alerts (SNS, GuardDuty)