How carmenda-nl/deidentify-engine works
Deidentify Engine is a FastAPI service for pseudonymizing care-organization text. A client submits a file, column mappings, and optionally a reusable datakey; a background worker loads and transforms the data, then clients poll or stream progress and retrieve a preview and metrics. The processing stages include datakey management and Deidentify model-based masking, with file I/O and progress/logging as supporting layers. README capabilities include REST access, high-performance Polars processing, and optional gateway authentication. Some endpoint and processing details are unsampled; edges reflect the excerpts, SOURCE INDEX references, or README.
Main entry points: run.py FastAPI application · process.py Process endpoint · processor.py Data processor · handler.py Deidentify handler
Components 18
HTTP API
- FastAPI application
app/run.py - Request schemas
app/api/schemas.py - App info endpoint
app/api/endpoints/app_info.py - Process endpoint
app/api/endpoints/process.py - Progress and results
app/api/endpoints/output.py - M2M security
app/api/utils/security.py - API file handling
app/api/utils/file_handling.py
Job execution
- Background worker
app/api/utils/worker.py - Progress tracker
app/core/utils/progress_tracker.py
Pseudonymization
- Data processor
app/core/processor.py - Datakey mapping
app/core/datakey.py - Deidentify handler
app/core/deidentify/handler.py - Model manager
app/core/deidentify/instance.py
Runtime support
- Data and key files
app/core/utils/file_handling.py - Job logging
app/core/utils/logger.py - Terminal output
app/core/utils/terminal.py - App configuration
app/main/config.py
Other components
- API client
Connections 28
- API client → FastAPI application: sends requests
README.md - FastAPI application → App info endpoint: routes to
- FastAPI application → Process endpoint: routes to
- FastAPI application → Progress and results: routes to
- Process endpoint → Request schemas: validates with
app/api/endpoints/process.py - Process endpoint → API file handling: uses
app/api/endpoints/process.py - Process endpoint → Background worker: submits job
app/api/endpoints/process.py - Process endpoint → Progress tracker: initializes
app/api/endpoints/process.py - Progress and results → Process endpoint: reads worker state
app/api/endpoints/output.py - Progress and results → Request schemas: returns schemas
app/api/endpoints/output.py - App info endpoint → Request schemas: returns schema
app/api/endpoints/app_info.py - FastAPI application → M2M security: registers security
- M2M security → App configuration: checks settings
app/api/utils/security.py - Background worker → Data processor: runs processing
app/api/utils/worker.py - Background worker → Progress tracker: updates status
app/api/utils/worker.py - Background worker → Job logging: attaches job log
app/api/utils/worker.py - Data processor → Progress tracker: reports progress
app/core/processor.py - Data processor → Data and key files: loads and saves
app/core/processor.py - Data processor → Datakey mapping: creates mapping
app/core/processor.py - Data processor → Deidentify handler: transforms data
app/core/processor.py - Datakey mapping → Data and key files: loads existing key
app/core/datakey.py - Deidentify handler → Model manager: creates tagger
app/core/deidentify/handler.py - Deidentify handler → Progress tracker: reports progress
app/core/deidentify/handler.py - Deidentify handler → Job logging: logs results
app/core/deidentify/handler.py - Deidentify handler → Terminal output: uses output
app/core/deidentify/handler.py - Model manager → App configuration: loads model setting
app/core/deidentify/instance.py - Data and key files → Progress tracker: updates loading progress
app/core/utils/file_handling.py - FastAPI application → App configuration: configures app
app/run.py