# carmenda-nl/deidentify-engine architecture

> Architecture diagram of the GitHub repository carmenda-nl/deidentify-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/carmenda-nl/deidentify-engine
- Repository: https://github.com/carmenda-nl/deidentify-engine

## Overview

Deidentify Engine is a FastAPI service for pseudonymizing care-organization text. A client submits a file, column mappings, and optionally a reusable datakey; a background worker loads and transforms the data, then clients poll or stream progress and retrieve a preview and metrics. The processing stages include datakey management and Deidentify model-based masking, with file I/O and progress/logging as supporting layers. README capabilities include REST access, high-performance Polars processing, and optional gateway authentication. Some endpoint and processing details are unsampled; edges reflect the excerpts, SOURCE INDEX references, or README.

Main entry points: [`app/run.py`](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/run.py) (FastAPI application), [`app/api/endpoints/process.py`](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/api/endpoints/process.py) (Process endpoint), [`app/core/processor.py`](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/processor.py) (Data processor), [`app/core/deidentify/handler.py`](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/deidentify/handler.py) (Deidentify handler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["HTTP API"]
  node_server["FastAPI application<br/>[run.py]"]
  node_request["Request schemas<br/>[schemas.py]"]
  node_info["App info endpoint<br/>[app_info.py]"]
  node_submit["Process endpoint<br/>[process.py]"]
  node_output["Progress and results<br/>[output.py]"]
  node_security["M2M security<br/>[security.py]"]
  node_api_files["API file handling<br/>[file_handling.py]"]
end

subgraph group_job["Job execution"]
  node_worker["Background worker<br/>[worker.py]"]
  node_tracker["Progress tracker"]
end

subgraph group_transform["Pseudonymization"]
  node_processor["Data processor<br/>[processor.py]"]
  node_datakey["Datakey mapping<br/>[datakey.py]"]
  node_handler["Deidentify handler<br/>[handler.py]"]
  node_model["Model manager<br/>[instance.py]"]
end

subgraph group_support["Runtime support"]
  node_files[("Data and key files<br/>[file_handling.py]")]
  node_logger["Job logging<br/>[logger.py]"]
  node_terminal["Terminal output<br/>[terminal.py]"]
  node_config["App configuration<br/>[config.py]"]
end

node_client(("API client"))

node_client -->|"sends requests"| node_server
node_server -->|"routes to"| node_info
node_server -->|"routes to"| node_submit
node_server -->|"routes to"| node_output
node_submit -->|"validates with"| node_request
node_submit -->|"uses"| node_api_files
node_submit -->|"submits job"| node_worker
node_submit -->|"initializes"| node_tracker
node_output -->|"reads worker state"| node_submit
node_output -->|"returns schemas"| node_request
node_info -->|"returns schema"| node_request
node_server -->|"registers security"| node_security
node_security -->|"checks settings"| node_config
node_worker -->|"runs processing"| node_processor
node_worker -->|"updates status"| node_tracker
node_worker -->|"attaches job log"| node_logger
node_processor -->|"reports progress"| node_tracker
node_processor -->|"loads and saves"| node_files
node_processor -->|"creates mapping"| node_datakey
node_processor -->|"transforms data"| node_handler
node_datakey -->|"loads existing key"| node_files
node_handler -->|"creates tagger"| node_model
node_handler -->|"reports progress"| node_tracker
node_handler -->|"logs results"| node_logger
node_handler -->|"uses output"| node_terminal
node_model -->|"loads model setting"| node_config
node_files -->|"updates loading progress"| node_tracker
node_server -->|"configures app"| node_config

click node_server "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/run.py"
click node_request "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/api/schemas.py"
click node_info "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/api/endpoints/app_info.py"
click node_submit "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/api/endpoints/process.py"
click node_output "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/api/endpoints/output.py"
click node_security "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/api/utils/security.py"
click node_api_files "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/api/utils/file_handling.py"
click node_worker "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/api/utils/worker.py"
click node_tracker "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/utils/progress_tracker.py"
click node_processor "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/processor.py"
click node_datakey "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/datakey.py"
click node_handler "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/deidentify/handler.py"
click node_model "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/deidentify/instance.py"
click node_files "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/utils/file_handling.py"
click node_logger "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/utils/logger.py"
click node_terminal "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/utils/terminal.py"
click node_config "https://github.com/carmenda-nl/deidentify-engine/blob/main/app/main/config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_server,node_request,node_info,node_submit,node_output,node_security,node_api_files,node_client toneBlue
class node_worker,node_tracker toneAmber
class node_processor,node_datakey,node_handler,node_model toneMint
class node_files,node_logger,node_terminal,node_config toneRose
```

## Components

### HTTP API

- [FastAPI application](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/run.py): `app/run.py`
- [Request schemas](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/api/schemas.py): `app/api/schemas.py`
- [App info endpoint](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/api/endpoints/app_info.py): `app/api/endpoints/app_info.py`
- [Process endpoint](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/api/endpoints/process.py): `app/api/endpoints/process.py`
- [Progress and results](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/api/endpoints/output.py): `app/api/endpoints/output.py`
- [M2M security](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/api/utils/security.py): `app/api/utils/security.py`
- [API file handling](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/api/utils/file_handling.py): `app/api/utils/file_handling.py`

### Job execution

- [Background worker](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/api/utils/worker.py): `app/api/utils/worker.py`
- [Progress tracker](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/utils/progress_tracker.py): `app/core/utils/progress_tracker.py`

### Pseudonymization

- [Data processor](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/processor.py): `app/core/processor.py`
- [Datakey mapping](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/datakey.py): `app/core/datakey.py`
- [Deidentify handler](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/deidentify/handler.py): `app/core/deidentify/handler.py`
- [Model manager](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/deidentify/instance.py): `app/core/deidentify/instance.py`

### Runtime support

- [Data and key files](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/utils/file_handling.py): `app/core/utils/file_handling.py`
- [Job logging](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/utils/logger.py): `app/core/utils/logger.py`
- [Terminal output](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/core/utils/terminal.py): `app/core/utils/terminal.py`
- [App configuration](https://github.com/carmenda-nl/deidentify-engine/blob/main/app/main/config.py): `app/main/config.py`

### Other components

- **API client**

## Connections

- API client → FastAPI application: sends requests (evidence: `README.md`)
- FastAPI application → App info endpoint: routes to
- FastAPI application → Process endpoint: routes to
- FastAPI application → Progress and results: routes to
- Process endpoint → Request schemas: validates with (evidence: `app/api/endpoints/process.py`)
- Process endpoint → API file handling: uses (evidence: `app/api/endpoints/process.py`)
- Process endpoint → Background worker: submits job (evidence: `app/api/endpoints/process.py`)
- Process endpoint → Progress tracker: initializes (evidence: `app/api/endpoints/process.py`)
- Progress and results → Process endpoint: reads worker state (evidence: `app/api/endpoints/output.py`)
- Progress and results → Request schemas: returns schemas (evidence: `app/api/endpoints/output.py`)
- App info endpoint → Request schemas: returns schema (evidence: `app/api/endpoints/app_info.py`)
- FastAPI application → M2M security: registers security
- M2M security → App configuration: checks settings (evidence: `app/api/utils/security.py`)
- Background worker → Data processor: runs processing (evidence: `app/api/utils/worker.py`)
- Background worker → Progress tracker: updates status (evidence: `app/api/utils/worker.py`)
- Background worker → Job logging: attaches job log (evidence: `app/api/utils/worker.py`)
- Data processor → Progress tracker: reports progress (evidence: `app/core/processor.py`)
- Data processor → Data and key files: loads and saves (evidence: `app/core/processor.py`)
- Data processor → Datakey mapping: creates mapping (evidence: `app/core/processor.py`)
- Data processor → Deidentify handler: transforms data (evidence: `app/core/processor.py`)
- Datakey mapping → Data and key files: loads existing key (evidence: `app/core/datakey.py`)
- Deidentify handler → Model manager: creates tagger (evidence: `app/core/deidentify/handler.py`)
- Deidentify handler → Progress tracker: reports progress (evidence: `app/core/deidentify/handler.py`)
- Deidentify handler → Job logging: logs results (evidence: `app/core/deidentify/handler.py`)
- Deidentify handler → Terminal output: uses output (evidence: `app/core/deidentify/handler.py`)
- Model manager → App configuration: loads model setting (evidence: `app/core/deidentify/instance.py`)
- Data and key files → Progress tracker: updates loading progress (evidence: `app/core/utils/file_handling.py`)
- FastAPI application → App configuration: configures app (evidence: `app/run.py`)