How nhsdigital/dtos-cohort-manager works
Project type and purpose
• This is a full-stack, cloud-native data-processing application (“CohortManager”) built on Azure.
• It ingests screening cohort files (via MESH), transforms and validates data, persists it, distributes cohorts, and exposes an administrative UI (Next.js).
File structure
• Top-level “application/CohortManager” contains:
• Set-up scripts (Docker, local database, CSV↔Parquet helpers)
• src/Functions – dozens of Azure Function Apps grouped by domain (CaasIntegration, CohortDistributionServices, DemographicServices, ScreeningValidationService, ServiceNowIntegration, ParticipantManagementServices, etc.)
• src/Web – Next.js React UI, API routes, end-to-end & unit tests
• “infrastructure” folder holds Terraform modules (tf-core and tf-audit) to provision Azure resources (Function Apps, Web App, SQL, Storage, Networking, Key Vault, App Insights).
• CI/CD definitions under “.azuredevops/pipelines” and GitHub Actions workflows automate build, test, image promotion, and deploy.
• “scripts” supports repository operations: Docker build & lint, Terraform workflows, static analysis, reporting.
• “tests” contains Unit, Integration, e2e (C#, Playwright), and Smoke tests.
README for additional insights
• The README is a generic repository template; it recommends using the C4 model for diagramming, documenting system interfaces, external dependencies, and integration points.
a. Main components (containers)
• User browser (UI) → Next.js Web App deployed to Azure App Service
• Azure Function Apps (grouped by domain)
• Azure Blob Storage (ingested files, transformations, error blobs)
• Azure Queue Storage (message queues for orchestrations)
• Azure SQL Database (cohort & participant data, exception logs)
• External services: MESH (file retrieval), PDS Demographic API, ServiceNow, any NHS webhooks
• DevOps: Azure DevOps/GitHub Actions pipelines, Terraform modules
b. Relationships & data flows
1. Operator uploads/places CAAS Parquet file → Blob Storage trigger on “receiveCaasFile” Function
2. “receiveCaasFile” enqueues batch messages → Durable Function for demographics enrichment via PDS
3. TransformDataService applies business rules → writes validated records to SQL and/or Exception blobs
4. CohortDistributionServices create and distribute cohort messages (writes to DB / external queues)
5. UI calls REST APIs (Next.js API routes) → functions in CohortDistributionServices or Shared DataServices
6. ExceptionHandling services write/read exception records → surfaced in UI
7. ServiceNowIntegration listens on Event Grid or queue → posts to ServiceNow
8. DevOps pipelines build Docker images, run tests, deploy Terraform to Azure
c. Architectural patterns & design principles
• Serverless microservices (Azure Function per business capability)
• Event-driven design (Blob triggers, Queue triggers, Durable Functions)
• Shared libraries (Common, Data, DataServices.Client/Core/Database, HealthChecks, Model) to enforce consistency
• Infrastructure as Code (Terraform), CI/CD pipelines (GitHub Actions + Azure DevOps YAML)
• C4 Model: Context → Containers → Components → Code
d. Key technologies & frameworks
• .NET 6/7 on Azure Functions (C#), Durable Functions
• Next.js 13 (React), TypeScript, Playwright, Jest
• Azure Blob & Queue Storage, Azure SQL Server
• Terraform, Azure DevOps, GitHub Actions, Docker, Make