How nekowanderer/dynamodb-migration-monitor works
Project Type and Purpose
• This is a Go‐based CLI tool (“dynamodb-migration-monitor”) that performs real-time monitoring and validation of an AWS DynamoDB table migration.
• It reads DynamoDB Streams from a source table, compares records against a target table (after S3 export/import + replication), and produces validation metrics.
Key Components (to include
a. AWS Infrastructure
• Source Account
• DynamoDB Source Table (with Streams enabled, NEW_IMAGES)
• S3 Bucket (for bulk export)
• Target Account
• DynamoDB Target Table
• Cross-Account IAM Roles/Permissions
• temp-ddb-migration-role (EC2 instance role) with permissions on both accounts
• Replication Script (external process that applies accumulated changes T1–T2)
b. Migration Mechanism
• S3 Export (bulk snapshot)
• S3 Import (bulk load to target table)
• Real-time Replication (via DynamoDB Streams & external script)
c. Validation Tool (running on EC2)
• CLI Frontend (cmd/datagen, cmd/datadel)
• Core Modules (internal/)
• cmd_flags.go (parses CLI args)
• dynamodb_clients.go (AWS SDK wrappers for Streams & Tables)
• stream_subscriber_v2.go (reads stream batches)
• stream_style_verification.go (per-event validation logic)
• Output & Metrics (console, logs)
Relationships and Data Flow
1. Bulk Migration:
• T1: Export source table → S3 → Import → target table
2. Real-Time Replication:
• Source Table → DynamoDB Streams → Replication Script → Target Table
3. Validation Tool:
• Reads from Source Table’s Stream
• For each record, queries Target Table (via AWS SDK)
• Applies sampling, retries, and batching (configurable buffers & channels)
• Emits metrics every N seconds (total/unique events, insert/modify counts, success rate)
Architectural Patterns & Principles
• Modular CLI (separate commands under cmd/, reusable logic in internal/)
• Asynchronous & Batch Processing (channels, buffer sizes)
• Retry & Delay Handling (handles DynamoDB eventual consistency)
• Cross-Account Security (IAM roles, least privilege)