How awslabs/pii-anonymizer works
This is an AWS-backed PII anonymization pipeline with a Streamlit upload and results interface. Its main workflow accepts files in S3, dispatches jobs through SQS and Step Functions, detects PII by file type, creates a shared synthetic mapping (or blackout mapping), redacts each file, and stores outputs and tracking data. Batch scheduling, audio processing, and workflow completion tracking are additional product paths. The README documents service integrations and stages; sampled source confirms key handlers, but not every processor-to-service call.
Main entry points: pii_detection_handler.py Detection handler · processors File processors · redact_handler.py Redaction handler · app.py Streamlit app
Components 22
Ingestion and orchestration
- Batch trigger
src/handlers/batch_trigger_handler.py - Job router
src/handlers/router_handler.py - Step Functions
PII processing
- Detection handler
src/handlers/pii_detection_handler.py - File processors
src/processors - PII detector
src/core/pii_detector.py - Synthetic handler
src/handlers/synthetic_handler.py - Synthetic generator
src/core/synthetic_pii_generator.py - Redaction handler
src/handlers/redact_handler.py - Redaction engines
src/redaction
Tracking and state
- Workflow tracker
src/handlers/workflow_tracker_handler.py - DynamoDB tables
src/infra/dynamodb_manager.py
User and AWS services
- Streamlit app
frontend/app.py - Input S3
- Output S3
- EventBridge
- SQS queue
- Textract OCR
- Transcribe
- Bedrock models
- Polly speech
Other components
- Uploader
Connections 30
- Uploader → Streamlit app: uploads files
- Streamlit app → Input S3: uploads to
- Input S3 → SQS queue: notifies
- EventBridge → Batch trigger: schedules scan
- Batch trigger → Input S3: scans files
- Batch trigger → SQS queue: queues jobs
- SQS queue → Job router: delivers jobs
- Job router → DynamoDB tables: checks and records jobs
- Job router → Step Functions: starts execution
- Step Functions → Detection handler: invokes per file
- Detection handler → File processors: dispatches by type
- File processors → PII detector: detects PII
- File processors → Textract OCR: OCR and coordinates
- File processors → Transcribe: transcribes audio
- Detection handler → Input S3: reads source files
- Detection handler → Output S3: writes detections
- Step Functions → Synthetic handler: invokes after detection
- Synthetic handler → Output S3: reads detections; writes mapping
- Synthetic handler → Synthetic generator: generates replacements
- Synthetic generator → Bedrock models: requests synthetic values
- Step Functions → Redaction handler: invokes after mapping
- Redaction handler → Output S3: reads detections and mapping
- Redaction handler → Redaction engines: applies replacements
- Redaction engines → Polly speech: synthesizes replacement speech
- Redaction handler → Output S3: stores redacted files
- Step Functions → EventBridge: completion event
- EventBridge → Workflow tracker: triggers completion tracking
- Workflow tracker → DynamoDB tables: updates status and concurrency
- Streamlit app → DynamoDB tables: reads job results
- Streamlit app → Output S3: downloads redacted output