How ministryofjustice/analytical-platform works
This repository isn’t a traditional full-stack application but rather a highly modular infrastructure repository for the Ministry of Justice Analytical Platform. Its purpose is twofold: to provide the technical documentation (and associated site building/publishing mechanisms) and to hold the infrastructure-as-code (IaC) artifacts—with heavy use of Terraform—to build and manage the analytical infrastructure used for storing, ingesting, and consuming data on AWS.
Main Components
• Documentation & Website Delivery:
– The source folder along with configuration files (e.g., config/tech-docs.yml) and Middleman preview scripts in the “scripts/docs” directory are responsible for rendering and publishing the technical documentation.
• CI/CD and Development Environment:
– The .github directory contains workflows, GitHub actions, dependency checks, and other automation tools for tasks like publishing documentation and enforcing code standards.
– The .devcontainer folder shows configuration for the development environment (e.g. VS Code remote containers).
• Infrastructure-as-Code Modules (Terraform):
– A significant part of the repository is taken up by the terraform directory. It contains multiple subdirectories representing different environments and components such as authentication (auth0), AWS infrastructure (EKS clusters, VPCs, security groups, EC2 instances, S3, Athena, CloudTrail, Route53), message brokers, build and artifact repositories, and more.
• Operational Scripts:
– The scripts folder encompasses various scripts that support operations like bucket archiving, user validation, data ingestion, Kubernetes operations, PagerDuty integrations, etc.
• Architectural Documentation & ADRs:
Relationships and Interactions
• CI/CD pipelines connect and coordinate the deployment of infrastructure (Terraform modules) and the documentation publishing process. Illustrate how GitHub Actions pull configurations from the repository and trigger deployment tasks.
• Operational scripts (for bucket archiving, ingress/egress tasks, and system monitoring) are run either as scheduled tasks or triggered by events in the CI/CD pipelines, and they interact with the AWS infrastructure.
Architectural Patterns and Design Principles
• Infrastructure as Code: The repository is organized around Terraform modules following the best practice of modular and environment-specific configuration.
• Separation of Concerns: Distinct directories are used to separate documentation, operational scripts, development environment configuration, and cloud infrastructure. This modular design eases maintenance and allows teams to work on different aspects independently.
• CI/CD Integration: GitHub Actions workflows are used to automate testing, deployment, validation, and publishing of both the documentation and infrastructure changes.
• Cloud-Native Design: The IAM, EKS, and various AWS-specific Terraform modules indicate a design that leverages AWS managed services and enforces security best practices (e.g., using Auth0 and AWS Secrets Manager for access).
Relevant Technologies, Frameworks, and Tools
• Terraform for Infrastructure Provisioning (with modules for AWS configuration, clusters, networking, IAM, etc.).
• GitHub Actions and related CI/CD tooling for automation of tests, validation, and deployments.
• GitHub’s infrastructure for repository management, dependabot, and pull request reviews.
• Docker and VS Code Dev Containers for local development environments.
• Middleman (or similar static site generator) for technical documentation rendering.
• AWS Services such as EKS, EC2, S3, IAM, CloudTrail, Athena, Route53, among others.
Essential Elements
• Clear Labels: Name each component clearly (e.g., “Documentation Server,” “CI/CD Pipeline,” “Terraform – AWS EKS Cluster,” “Operational Scripts for Ingestion,” etc.).
• Annotation of Technologies: Where relevant, annotate components with the key technologies (e.g., “Terraform,” “AWS EKS,” “GitHub Actions,” “Middleman”) to offer a quick technical reference.
Happy diagramming!