# nhsdigital/dtos-cohort-manager architecture

> Architecture diagram of the GitHub repository nhsdigital/dtos-cohort-manager, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nhsdigital/dtos-cohort-manager
- Repository: https://github.com/nhsdigital/dtos-cohort-manager

## Overview

##### Project type and purpose
- This is a full-stack, cloud-native data-processing application (“CohortManager”) built on Azure.
- It ingests screening cohort files (via MESH), transforms and validates data, persists it, distributes cohorts, and exposes an administrative UI (Next.js).

##### File structure
- Top-level “application/CohortManager” contains:
- Set-up scripts (Docker, local database, CSV↔Parquet helpers)
- src/Functions – dozens of Azure Function Apps grouped by domain (CaasIntegration, CohortDistributionServices, DemographicServices, ScreeningValidationService, ServiceNowIntegration, ParticipantManagementServices, etc.)
- src/Web – Next.js React UI, API routes, end-to-end & unit tests
- “infrastructure” folder holds Terraform modules (tf-core and tf-audit) to provision Azure resources (Function Apps, Web App, SQL, Storage, Networking, Key Vault, App Insights).
- CI/CD definitions under “.azuredevops/pipelines” and GitHub Actions workflows automate build, test, image promotion, and deploy.
- “scripts” supports repository operations: Docker build & lint, Terraform workflows, static analysis, reporting.
- “tests” contains Unit, Integration, e2e (C#, Playwright), and Smoke tests.

##### README for additional insights
- The README is a generic repository template; it recommends using the C4 model for diagramming, documenting system interfaces, external dependencies, and integration points.

a. Main components (containers)
- User browser (UI) → Next.js Web App deployed to Azure App Service
- Azure Function Apps (grouped by domain)
- Azure Blob Storage (ingested files, transformations, error blobs)
- Azure Queue Storage (message queues for orchestrations)
- Azure SQL Database (cohort & participant data, exception logs)
- External services: MESH (file retrieval), PDS Demographic API, ServiceNow, any NHS webhooks
- DevOps: Azure DevOps/GitHub Actions pipelines, Terraform modules
b. Relationships & data flows
1. Operator uploads/places CAAS Parquet file → Blob Storage trigger on “receiveCaasFile” Function
2. “receiveCaasFile” enqueues batch messages → Durable Function for demographics enrichment via PDS
3. TransformDataService applies business rules → writes validated records to SQL and/or Exception blobs
4. CohortDistributionServices create and distribute cohort messages (writes to DB / external queues)
5. UI calls REST APIs (Next.js API routes) → functions in CohortDistributionServices or Shared DataServices
6. ExceptionHandling services write/read exception records → surfaced in UI
7. ServiceNowIntegration listens on Event Grid or queue → posts to ServiceNow
8. DevOps pipelines build Docker images, run tests, deploy Terraform to Azure
c. Architectural patterns & design principles
- Serverless microservices (Azure Function per business capability)
- Event-driven design (Blob triggers, Queue triggers, Durable Functions)
- Shared libraries (Common, Data, DataServices.Client/Core/Database, HealthChecks, Model) to enforce consistency
- Infrastructure as Code (Terraform), CI/CD pipelines (GitHub Actions + Azure DevOps YAML)
- C4 Model: Context → Containers → Components → Code
d. Key technologies & frameworks
- .NET 6/7 on Azure Functions (C#), Durable Functions
- Next.js 13 (React), TypeScript, Playwright, Jest
- Azure Blob & Queue Storage, Azure SQL Server
- Terraform, Azure DevOps, GitHub Actions, Docker, Make

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Services
    subgraph "External Services"
        direction TB
        MESH["MESH"]:::external
        PDS["PDS Demographic API"]:::external
        SNOW["ServiceNow"]:::external
    end

    %% Storage Services
    subgraph "Azure Storage & DB"
        direction TB
        Blob["Azure Blob Storage"]:::azurePaaS
        Queue["Azure Queue Storage"]:::azurePaaS
        SQLDB["Azure SQL Database"]:::azurePaaS
    end

    %% Shared Libraries
    Shared["Shared Libraries"]:::functions
    click Shared "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/Shared"

    %% Function Apps
    subgraph "Azure Function Apps" 
        direction TB

        subgraph "CaasIntegration" 
            direction TB
            ReceiveCaas["receiveCaasFile"]:::functions
            click ReceiveCaas "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/CaasIntegration/receiveCaasFile"
            RetrieveMesh["RetrieveMeshFile"]:::functions
            click RetrieveMesh "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/CaasIntegration/RetrieveMeshFile"
        end

        subgraph "DemographicServices"
            direction TB
            DemMgmt["DemographicDataManagementFunction"]:::functions
            click DemMgmt "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/DemographicServices/DemographicDataManagementFunction"
            DemDurable["DemographicDurableFunction"]:::functions
            click DemDurable "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/DemographicServices/DemographicDurableFunction"
            RetrievePDS["RetrievePDSDemographic"]:::functions
            click RetrievePDS "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/DemographicServices/RetrievePDSDemographic"
            NEMSSub["NEMSSubscribe"]:::functions
            click NEMSSub "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/DemographicServices/NEMSSubscribe"
            NEMSUnsub["NEMSUnSubscription"]:::functions
            click NEMSUnsub "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/DemographicServices/NEMSUnSubscription"
        end

        subgraph "CohortDistributionServices"
            direction TB
            TransformData["TransformDataService"]:::functions
            click TransformData "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/CohortDistributionServices/TransformDataService"
            AddDist["AddCohortDistributionData"]:::functions
            click AddDist "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/CohortDistributionServices/AddCohortDistributionData"
            CreateDist["CreateCohortDistribution"]:::functions
            click CreateDist "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/CohortDistributionServices/CreateCohortDistribution"
            RetrieveDist["RetrieveCohortDistribution"]:::functions
            click RetrieveDist "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/CohortDistributionServices/RetrieveCohortDistribution"
            AuditDist["RetrieveCohortRequestAudit"]:::functions
            click AuditDist "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/CohortDistributionServices/RetrieveCohortRequestAudit"
            PartData["RetrieveParticipantData"]:::functions
            click PartData "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/CohortDistributionServices/RetrieveParticipantData"
            SPA["ServiceProviderAllocationService"]:::functions
            click SPA "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/CohortDistributionServices/ServiceProviderAllocationService"
            ValidateDist["ValidateCohortDistributionRecord"]:::functions
            click ValidateDist "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/CohortDistributionServices/ValidateCohortDistributionRecord"
        end

        ParticipantMgmt["ParticipantManagementServices"]:::functions
        click ParticipantMgmt "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/ParticipantManagementServices"

        ScreeningVal["ScreeningValidationService"]:::functions
        click ScreeningVal "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/ScreeningValidationService"

        ExceptionHandle["ExceptionHandling"]:::functions
        click ExceptionHandle "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/ExceptionHandling"

        SNIntegration["ServiceNowIntegrationService"]:::functions
        click SNIntegration "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Functions/ServiceNowIntegrationService"
    end

    %% Web UI
    WebUI["Web UI (Next.js)"]:::web
    click WebUI "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/application/CohortManager/src/Web"

    %% Infrastructure as Code
    subgraph "Infrastructure (Terraform)" 
        direction TB
        TFCore["tf-core"]:::infra
        click TFCore "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/infrastructure/tf-core"
        TFAudit["tf-audit"]:::infra
        click TFAudit "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/infrastructure/tf-audit"
    end

    %% CI/CD
    subgraph "CI/CD Pipelines" 
        direction TB
        DevOps["Azure DevOps Pipelines"]:::infra
        click DevOps "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/.azuredevops/pipelines"
        GHActions["GitHub Actions"]:::infra
        click GHActions "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/.github/workflows"
        Scripts["Automation Scripts"]:::infra
        click Scripts "https://github.com/nhsdigital/dtos-cohort-manager/tree/main/scripts"
    end

    %% Data Flow
    MESH -->|Parquet file| RetrieveMesh
    RetrieveMesh -->|BlobTrigger| Blob
    Blob -->|trigger| ReceiveCaas
    ReceiveCaas -->|enqueue| Queue
    Queue -->|trigger| DemDurable
    DemDurable -->|call| RetrievePDS
    RetrievePDS -->|data| DemMgmt
    DemMgmt -->|validated| Queue
    Queue -->|trigger| TransformData
    TransformData -->|write| SQLDB
    TransformData -->|error blobs| Blob
    TransformData -->|enqueue| Queue
    Queue -->|trigger| AddDist
    AddDist --> CreateDist --> ValidateDist --> ServiceProviderAllocationService
    ServiceProviderAllocationService --> AddDist
    CreateDist -->|records| SQLDB
    ValidateDist -->|invalid| ExceptionHandle
    ExceptionHandle -->|error records| SQLDB
    CreateDist -->|messages| Queue
    Queue -->|trigger| PartData
    PartData -->|data| SQLDB

    %% UI interactions
    WebUI -->|REST API| RetrieveDist
    RetrieveDist --> SQLDB
    WebUI -->|exceptions| ExceptionHandle
    WebUI -->|participant CRUD| ParticipantMgmt

    %% ServiceNow
    SNIntegration -->|Event Grid/Queue| SNOW

    %% Shared library usage
    Shared --> ReceiveCaas
    Shared --> RetrieveMesh
    Shared --> DemMgmt
    Shared --> DemDurable
    Shared --> RetrievePDS
    Shared --> TransformData
    Shared --> AddDist
    Shared --> CreateDist
    Shared --> RetrieveDist
    Shared --> ParticipantMgmt
    Shared --> ScreeningVal
    Shared --> ExceptionHandle
    Shared --> SNIntegration

    %% Infra links
    TFCore --> ReceiveCaas
    TFCore --> RetrieveMesh
    TFCore --> DemMgmt
    TFCore --> TransformData
    TFCore --> WebUI
    TFCore --> Blob
    TFCore --> Queue
    TFCore --> SQLDB
    TFAudit -->|monitors| TFCore

    %% CI/CD links
    DevOps --> TFCore
    DevOps --> WebUI
    DevOps --> ReceiveCaas
    GHActions --> TFCore
    GHActions --> WebUI
    GHActions --> ReceiveCaas
    Scripts --> DevOps

    %% Styles
    classDef azurePaaS fill:#dae8fc,stroke:#6c8ebf
    classDef functions fill:#e1f5d1,stroke:#82b366
    classDef web fill:#fff2cc,stroke:#d6b656
    classDef external fill:#f5f5f5,stroke:#999
    classDef infra fill:#ffe6cc,stroke:#d79b00
```
