# nekowanderer/dynamodb-migration-monitor architecture

> Architecture diagram of the GitHub repository nekowanderer/dynamodb-migration-monitor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nekowanderer/dynamodb-migration-monitor
- Repository: https://github.com/nekowanderer/dynamodb-migration-monitor

## Overview

##### Project Type and Purpose
- This is a Go‐based CLI tool (“dynamodb-migration-monitor”) that performs real-time monitoring and validation of an AWS DynamoDB table migration.
- It reads DynamoDB Streams from a source table, compares records against a target table (after S3 export/import + replication), and produces validation metrics.

##### Key Components (to include
a. AWS Infrastructure
- Source Account
- DynamoDB Source Table (with Streams enabled, NEW_IMAGES)
- S3 Bucket (for bulk export)
- Target Account
- DynamoDB Target Table
- Cross-Account IAM Roles/Permissions
- temp-ddb-migration-role (EC2 instance role) with permissions on both accounts
- Replication Script (external process that applies accumulated changes T1–T2)
b. Migration Mechanism
- S3 Export (bulk snapshot)
- S3 Import (bulk load to target table)
- Real-time Replication (via DynamoDB Streams & external script)
c. Validation Tool (running on EC2)
- CLI Frontend (cmd/datagen, cmd/datadel)
- Core Modules (internal/)
- cmd_flags.go (parses CLI args)
- dynamodb_clients.go (AWS SDK wrappers for Streams & Tables)
- stream_subscriber_v2.go (reads stream batches)
- stream_style_verification.go (per-event validation logic)
- Output & Metrics (console, logs)

##### Relationships and Data Flow
1. Bulk Migration:
- T1: Export source table → S3 → Import → target table
2. Real-Time Replication:
- Source Table → DynamoDB Streams → Replication Script → Target Table
3. Validation Tool:
- Reads from Source Table’s Stream
- For each record, queries Target Table (via AWS SDK)
- Applies sampling, retries, and batching (configurable buffers & channels)
- Emits metrics every N seconds (total/unique events, insert/modify counts, success rate)

##### Architectural Patterns & Principles
- Modular CLI (separate commands under cmd/, reusable logic in internal/)
- Asynchronous & Batch Processing (channels, buffer sizes)
- Retry & Delay Handling (handles DynamoDB eventual consistency)
- Cross-Account Security (IAM roles, least privilege)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Title
    title["DynamoDB Migration Monitor Architecture"]

    %% AWS Infrastructure Swimlanes
    subgraph "AWS Accounts"
        direction TB
        subgraph "Source Account" 
            direction TB
            DynamoDB_Source["DynamoDB Source Table\n(Streams: NEW_IMAGES)"]:::aws
            S3_Bucket["S3 Bucket\n(bulk export)"]:::aws
            Streams_Source["DynamoDB Stream"]:::aws
        end
        subgraph "Target Account"
            direction TB
            DynamoDB_Target["DynamoDB Target Table"]:::aws
        end
        IAM_Role["temp-ddb-migration-role\n(Cross-Account IAM Role)"]:::security
    end

    %% EC2 Validation Tool
    subgraph "EC2 Instance\nValidation Tool" 
        direction TB
        main_go["main.go\n(CLI Entrypoint)"]:::cli
        cmd_datagen["datagen\n(cmd/datagen/main.go)"]:::cli
        cmd_datadel["datadel\n(cmd/datadel/main.go)"]:::cli
        cmd_flags["cmd_flags.go"]:::core
        dynamodb_clients["dynamodb_clients.go"]:::core
        stream_subscriber["stream_subscriber_v2.go"]:::core
        stream_verifier["stream_style_verification.go"]:::core
        Metrics["Metrics & Logging\n(console, logs)"]:::output
    end

    %% External Replication Script
    Replication_Script["Replication Script\n(real-time apply)"]:::external

    %% Data Flows - Bulk Migration
    DynamoDB_Source -->|"T1: Export"| S3_Bucket
    S3_Bucket -->|"T2: Import"| DynamoDB_Target

    %% Real-Time Replication Flow
    DynamoDB_Source --> Streams_Source
    Streams_Source -->|"T3: Stream events"| Replication_Script
    Replication_Script -->|"T4: Apply changes"| DynamoDB_Target

    %% Validation Tool Flow
    Streams_Source -->|"Stream events"| main_go
    main_go --> cmd_flags
    cmd_flags --> dynamodb_clients
    dynamodb_clients --> stream_subscriber
    stream_subscriber --> stream_verifier
    stream_verifier --> Metrics
    stream_verifier -->|"Query target\n(wait & retry)"| dynamodb_clients
    dynamodb_clients -->|"GetItem/Query"| DynamoDB_Target

    %% IAM Role connections
    IAM_Role --- EC2_Instance(("EC2 Instance")):::infra
    IAM_Role --- DynamoDB_Source
    IAM_Role --- DynamoDB_Target

    %% Click Events
    click main_go "https://github.com/nekowanderer/dynamodb-migration-monitor/blob/main//main.go"
    click cmd_datagen "https://github.com/nekowanderer/dynamodb-migration-monitor/blob/main//cmd/datagen/main.go"
    click cmd_datadel "https://github.com/nekowanderer/dynamodb-migration-monitor/blob/main//cmd/datadel/main.go"
    click cmd_flags "https://github.com/nekowanderer/dynamodb-migration-monitor/blob/main//internal/cmd_flags.go"
    click dynamodb_clients "https://github.com/nekowanderer/dynamodb-migration-monitor/blob/main//internal/dynamodb_clients.go"
    click stream_subscriber "https://github.com/nekowanderer/dynamodb-migration-monitor/blob/main//internal/stream_subscriber_v2.go"
    click stream_verifier "https://github.com/nekowanderer/dynamodb-migration-monitor/blob/main//internal/stream_style_verification.go"
    click README["README.md"] "/README.md"
    click go_mod["go.mod"] "/go.mod"
    click go_sum["go.sum"] "/go.sum"
    click monitoring_spec["monitoring_mechanism.md"] "/monitoring_mechanism.md"
    click ci_pipeline[".github/workflows/go.yml"] "/.github/workflows/go.yml"
    click test_keys["test_keys.csv"] "/test_keys.csv"

    %% Styles
    classDef aws fill:#F8C471,stroke:#E67E22,color:#000
    classDef infra fill:#ABEBC6,stroke:#27AE60,color:#000
    classDef cli fill:#A9DFBF,stroke:#229954,color:#000
    classDef core fill:#D6EAF8,stroke:#5DADE2,color:#000
    classDef external fill:#D2B4DE,stroke:#8E44AD,color:#000
    classDef security fill:#F5B7B1,stroke:#C0392B,color:#000
    classDef output fill:#FCF3CF,stroke:#F1C40F,color:#000
    class main_go,cmd_datagen,cmd_datadel,cmd_flags,dynamodb_clients,stream_subscriber,stream_verifier cli
    class DynamoDB_Source,S3_Bucket,DynamoDB_Target,Streams_Source aws
    class Replication_Script external
    class IAM_Role security
    class EC2_Instance infra
    class Metrics output
```
