# awslabs/pii-anonymizer architecture

> Architecture diagram of the GitHub repository awslabs/pii-anonymizer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/awslabs/pii-anonymizer
- Repository: https://github.com/awslabs/pii-anonymizer
- Video tour (about a minute): https://gitdiagram.com/awslabs/pii-anonymizer/video

## Overview

This is an AWS-backed PII anonymization pipeline with a Streamlit upload and results interface. Its main workflow accepts files in S3, dispatches jobs through SQS and Step Functions, detects PII by file type, creates a shared synthetic mapping (or blackout mapping), redacts each file, and stores outputs and tracking data. Batch scheduling, audio processing, and workflow completion tracking are additional product paths. The README documents service integrations and stages; sampled source confirms key handlers, but not every processor-to-service call.

Main entry points: [`src/handlers/pii_detection_handler.py`](https://github.com/awslabs/pii-anonymizer/blob/main/src/handlers/pii_detection_handler.py) (Detection handler), [`src/processors`](https://github.com/awslabs/pii-anonymizer/tree/main/src/processors) (File processors), [`src/handlers/redact_handler.py`](https://github.com/awslabs/pii-anonymizer/blob/main/src/handlers/redact_handler.py) (Redaction handler), [`frontend/app.py`](https://github.com/awslabs/pii-anonymizer/blob/main/frontend/app.py) (Streamlit app)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion and orchestration"]
  node_batch["Batch trigger"]
  node_router["Job router<br/>[router_handler.py]"]
  node_stepfunctions["Step Functions"]
end

subgraph group_processing["PII processing"]
  node_detection["Detection handler"]
  node_processors["File processors"]
  node_detector["PII detector<br/>[pii_detector.py]"]
  node_synthetic["Synthetic handler"]
  node_generator["Synthetic generator"]
  node_redact["Redaction handler<br/>[redact_handler.py]"]
  node_redactor["Redaction engines"]
end

subgraph group_state["Tracking and state"]
  node_tracker["Workflow tracker"]
  node_dynamodb[("DynamoDB tables")]
end

subgraph group_interfaces["User and AWS services"]
  node_frontend["Streamlit app<br/>[app.py]"]
  node_s3input[("Input S3")]
  node_s3output[("Output S3")]
  node_eventbridge["EventBridge"]
  node_sqs["SQS queue"]
  node_textract["Textract OCR"]
  node_transcribe["Transcribe"]
  node_bedrock["Bedrock models"]
  node_polly["Polly speech"]
end

node_user(("Uploader"))

node_user -->|"uploads files"| node_frontend
node_frontend -->|"uploads to"| node_s3input
node_s3input -->|"notifies"| node_sqs
node_eventbridge -.->|"schedules scan"| node_batch
node_batch -.->|"scans files"| node_s3input
node_batch -.->|"queues jobs"| node_sqs
node_sqs -->|"delivers jobs"| node_router
node_router -->|"checks and records jobs"| node_dynamodb
node_router -->|"starts execution"| node_stepfunctions
node_stepfunctions -->|"invokes per file"| node_detection
node_detection -->|"dispatches by type"| node_processors
node_processors -->|"detects PII"| node_detector
node_processors -.->|"OCR and coordinates"| node_textract
node_processors -.->|"transcribes audio"| node_transcribe
node_detection -->|"reads source files"| node_s3input
node_detection -->|"writes detections"| node_s3output
node_stepfunctions -->|"invokes after detection"| node_synthetic
node_synthetic -->|"reads detections; writes mapping"| node_s3output
node_synthetic -->|"generates replacements"| node_generator
node_generator -.->|"requests synthetic values"| node_bedrock
node_stepfunctions -->|"invokes after mapping"| node_redact
node_redact -->|"reads detections and mapping"| node_s3output
node_redact -->|"applies replacements"| node_redactor
node_redactor -.->|"synthesizes replacement speech"| node_polly
node_redact -->|"stores redacted files"| node_s3output
node_stepfunctions -->|"completion event"| node_eventbridge
node_eventbridge -->|"triggers completion tracking"| node_tracker
node_tracker -->|"updates status and concurrency"| node_dynamodb
node_frontend -->|"reads job results"| node_dynamodb
node_frontend -->|"downloads redacted output"| node_s3output

click node_frontend "https://github.com/awslabs/pii-anonymizer/blob/main/frontend/app.py"
click node_batch "https://github.com/awslabs/pii-anonymizer/blob/main/src/handlers/batch_trigger_handler.py"
click node_router "https://github.com/awslabs/pii-anonymizer/blob/main/src/handlers/router_handler.py"
click node_detection "https://github.com/awslabs/pii-anonymizer/blob/main/src/handlers/pii_detection_handler.py"
click node_processors "https://github.com/awslabs/pii-anonymizer/tree/main/src/processors"
click node_detector "https://github.com/awslabs/pii-anonymizer/blob/main/src/core/pii_detector.py"
click node_synthetic "https://github.com/awslabs/pii-anonymizer/blob/main/src/handlers/synthetic_handler.py"
click node_generator "https://github.com/awslabs/pii-anonymizer/blob/main/src/core/synthetic_pii_generator.py"
click node_redact "https://github.com/awslabs/pii-anonymizer/blob/main/src/handlers/redact_handler.py"
click node_redactor "https://github.com/awslabs/pii-anonymizer/tree/main/src/redaction"
click node_tracker "https://github.com/awslabs/pii-anonymizer/blob/main/src/handlers/workflow_tracker_handler.py"
click node_dynamodb "https://github.com/awslabs/pii-anonymizer/blob/main/src/infra/dynamodb_manager.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_batch,node_router,node_stepfunctions toneBlue
class node_detection,node_processors,node_detector,node_synthetic,node_generator,node_redact,node_redactor toneAmber
class node_tracker,node_dynamodb toneMint
class node_frontend,node_s3input,node_s3output,node_eventbridge,node_sqs,node_textract,node_transcribe,node_bedrock,node_polly toneRose
class node_user toneIndigo
```

## Components

### Ingestion and orchestration

- [Batch trigger](https://github.com/awslabs/pii-anonymizer/blob/main/src/handlers/batch_trigger_handler.py): `src/handlers/batch_trigger_handler.py`
- [Job router](https://github.com/awslabs/pii-anonymizer/blob/main/src/handlers/router_handler.py): `src/handlers/router_handler.py`
- **Step Functions**

### PII processing

- [Detection handler](https://github.com/awslabs/pii-anonymizer/blob/main/src/handlers/pii_detection_handler.py): `src/handlers/pii_detection_handler.py`
- [File processors](https://github.com/awslabs/pii-anonymizer/tree/main/src/processors): `src/processors`
- [PII detector](https://github.com/awslabs/pii-anonymizer/blob/main/src/core/pii_detector.py): `src/core/pii_detector.py`
- [Synthetic handler](https://github.com/awslabs/pii-anonymizer/blob/main/src/handlers/synthetic_handler.py): `src/handlers/synthetic_handler.py`
- [Synthetic generator](https://github.com/awslabs/pii-anonymizer/blob/main/src/core/synthetic_pii_generator.py): `src/core/synthetic_pii_generator.py`
- [Redaction handler](https://github.com/awslabs/pii-anonymizer/blob/main/src/handlers/redact_handler.py): `src/handlers/redact_handler.py`
- [Redaction engines](https://github.com/awslabs/pii-anonymizer/tree/main/src/redaction): `src/redaction`

### Tracking and state

- [Workflow tracker](https://github.com/awslabs/pii-anonymizer/blob/main/src/handlers/workflow_tracker_handler.py): `src/handlers/workflow_tracker_handler.py`
- [DynamoDB tables](https://github.com/awslabs/pii-anonymizer/blob/main/src/infra/dynamodb_manager.py): `src/infra/dynamodb_manager.py`

### User and AWS services

- [Streamlit app](https://github.com/awslabs/pii-anonymizer/blob/main/frontend/app.py): `frontend/app.py`
- **Input S3**
- **Output S3**
- **EventBridge**
- **SQS queue**
- **Textract OCR**
- **Transcribe**
- **Bedrock models**
- **Polly speech**

### Other components

- **Uploader**

## Connections

- Uploader → Streamlit app: uploads files
- Streamlit app → Input S3: uploads to
- Input S3 → SQS queue: notifies
- EventBridge → Batch trigger: schedules scan
- Batch trigger → Input S3: scans files
- Batch trigger → SQS queue: queues jobs
- SQS queue → Job router: delivers jobs
- Job router → DynamoDB tables: checks and records jobs
- Job router → Step Functions: starts execution
- Step Functions → Detection handler: invokes per file
- Detection handler → File processors: dispatches by type
- File processors → PII detector: detects PII
- File processors → Textract OCR: OCR and coordinates
- File processors → Transcribe: transcribes audio
- Detection handler → Input S3: reads source files
- Detection handler → Output S3: writes detections
- Step Functions → Synthetic handler: invokes after detection
- Synthetic handler → Output S3: reads detections; writes mapping
- Synthetic handler → Synthetic generator: generates replacements
- Synthetic generator → Bedrock models: requests synthetic values
- Step Functions → Redaction handler: invokes after mapping
- Redaction handler → Output S3: reads detections and mapping
- Redaction handler → Redaction engines: applies replacements
- Redaction engines → Polly speech: synthesizes replacement speech
- Redaction handler → Output S3: stores redacted files
- Step Functions → EventBridge: completion event
- EventBridge → Workflow tracker: triggers completion tracking
- Workflow tracker → DynamoDB tables: updates status and concurrency
- Streamlit app → DynamoDB tables: reads job results
- Streamlit app → Output S3: downloads redacted output