# singer-io/tap-dynamodb architecture

> Architecture diagram of the GitHub repository singer-io/tap-dynamodb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/singer-io/tap-dynamodb
- Repository: https://github.com/singer-io/tap-dynamodb

## Overview

##### Project type and purpose
- This is an open-source “Singer” tap whose sole purpose is to extract data from AWS DynamoDB tables (either via full-table scans or DynamoDB Streams) and emit it as JSON-formatted Singer messages on stdout. It is not a web app or microservice but a CLI data-export utility.

##### File structure
- bin/test-db: CLI entrypoint for local testing.
- setup.py: installation and entry-point definition (“tap-dynamodb” console script).
- tap_dynamodb/: core package
- __init__.py
- discover.py: schema discovery logic
- dynamodb.py: thin AWS DynamoDB client wrapper (uses boto3)
- deserialize.py: convert DynamoDB wire format to plain JSON
- sync.py: orchestrates syncing data
- sync_strategies/: two interchangeable sync strategies
– full_table.py
– log_based.py
- tests/: full integration tests for discovery and both sync strategies, plus unit tests for deserializer and helper functions.

a. Main components
- CLI Layer (“tap-dynamodb” binary)
- Configuration Loader/State Manager (reads JSON config + state file)
- Core Controller (sync.py)
- Discovery Module (discover.py)
- AWS DynamoDB Client (dynamodb.py → boto3)
- Sync Strategies (full_table & log_based)
- Deserializer (deserialize.py)
- Output Formatter (Singer message builder)
- stdout (Sink for JSON records)

b. Relationships & data flow
1. User invokes CLI → loads config + state
2. CLI calls controller in sync.py
3. Controller either runs discovery or chosen sync strategy
4. Sync strategy uses dynamodb.py to query/scan table or stream
5. Raw AWS responses flow into deserialize.py
6. Deserialized rows go into Singer message builder
7. Messages emitted to stdout

c. Architectural patterns & principles
- Plugin strategy pattern: sync_strategies folder holds interchangeable sync algorithms.
- Single Responsibility: each module has one focused job (discover vs sync vs deserialize).
- Decoupling: AWS interactions are isolated in a wrapper module.
- Testable design: modules can be unit-tested in isolation.

d. Key technologies & libraries
- Python 3 CLI
- boto3 (AWS SDK)
- Singer specification (JSON message schemas)
- pytest for testing

##### Final advice
- Sync_strategies under a “Strategy” container.
- Don’t forget the discovery path (discover.py → schema messages).
- Config & state file as inputs to the system.
- Aim for high detail: separate deserializer from output formatter, highlight AWS wrapper, and show the pluggable strategies clearly.

## Diagram (Mermaid)

```mermaid
flowchart TB
  subgraph "I/O" 
    CLI_Test["bin/test-db"]:::io
    CLI_Script["setup.py"]:::io
    ConfigFile["config.json"]:::io
    StateFile["state.json"]:::io
    STDOUT["stdout"]:::io
  end

  subgraph "Core Modules"
    Sync["Sync Controller / Config Loader"]:::core
    Discover["Discovery Module"]:::core
    Deserial["Deserializer"]:::core
    Formatter["Output Formatter"]:::core
    subgraph "Sync Strategies"
      Full["Full-Table Strategy"]:::plugin
      Log["Log-Based Strategy"]:::plugin
    end
  end

  DynamoWrapper["AWS DynamoDB Client"]:::core
  AWS["AWS DynamoDB"]:::external
  Tests["Tests"]:::test

  CLI_Test -->|"invoke"| Sync
  CLI_Script -->|"invoke"| Sync
  ConfigFile -->|"load config"| Sync
  StateFile -->|"load state"| Sync

  Sync -->|"run discovery"| Discover
  Sync -->|"run strategy"| Full
  Sync -->|"run strategy"| Log

  Full -->|"scan/query"| DynamoWrapper
  Log -->|"stream read"| DynamoWrapper

  DynamoWrapper -->|"AWS SDK calls"| AWS
  AWS -->|"raw response"| Deserial
  Deserial -->|"JSON records"| Formatter
  Discover -->|"schema messages"| Formatter
  Formatter -->|"Singer messages"| STDOUT

  classDef external fill:#AED6F1,stroke:#21618C,color:#1B4F72;
  classDef core fill:#A9DFBF,stroke:#196F3D,color:#145A32;
  classDef io fill:#F9E79F,stroke:#B9770E,color:#7D6608;
  classDef plugin fill:#F5CBA7,stroke:#B9770E,color:#784212;
  classDef test fill:#D7DBDD,stroke:#7B7D7D,color:#424949;

  click CLI_Test "https://github.com/singer-io/tap-dynamodb/tree/master/bin/test-db"
  click CLI_Script "https://github.com/singer-io/tap-dynamodb/blob/master/setup.py"
  click Sync "https://github.com/singer-io/tap-dynamodb/blob/master/tap_dynamodb/sync.py"
  click Discover "https://github.com/singer-io/tap-dynamodb/blob/master/tap_dynamodb/discover.py"
  click DynamoWrapper "https://github.com/singer-io/tap-dynamodb/blob/master/tap_dynamodb/dynamodb.py"
  click Deserial "https://github.com/singer-io/tap-dynamodb/blob/master/tap_dynamodb/deserialize.py"
  click Full "https://github.com/singer-io/tap-dynamodb/blob/master/tap_dynamodb/sync_strategies/full_table.py"
  click Log "https://github.com/singer-io/tap-dynamodb/blob/master/tap_dynamodb/sync_strategies/log_based.py"
  click Tests "https://github.com/singer-io/tap-dynamodb/tree/master/tests"
```
