How singer-io/tap-dynamodb works
Project type and purpose
• This is an open-source “Singer” tap whose sole purpose is to extract data from AWS DynamoDB tables (either via full-table scans or DynamoDB Streams) and emit it as JSON-formatted Singer messages on stdout. It is not a web app or microservice but a CLI data-export utility.
File structure
• bin/test-db: CLI entrypoint for local testing.
• setup.py: installation and entry-point definition (“tap-dynamodb” console script).
• tap_dynamodb/: core package
• __init__.py
• discover.py: schema discovery logic
• dynamodb.py: thin AWS DynamoDB client wrapper (uses boto3)
• deserialize.py: convert DynamoDB wire format to plain JSON
• sync.py: orchestrates syncing data
• sync_strategies/: two interchangeable sync strategies
– full_table.py
– log_based.py
• tests/: full integration tests for discovery and both sync strategies, plus unit tests for deserializer and helper functions.
a. Main components
• CLI Layer (“tap-dynamodb” binary)
• Configuration Loader/State Manager (reads JSON config + state file)
• Core Controller (sync.py)
• Discovery Module (discover.py)
• AWS DynamoDB Client (dynamodb.py → boto3)
• Sync Strategies (full_table & log_based)
• Deserializer (deserialize.py)
• Output Formatter (Singer message builder)
• stdout (Sink for JSON records)
b. Relationships & data flow
1. User invokes CLI → loads config + state
2. CLI calls controller in sync.py
3. Controller either runs discovery or chosen sync strategy
4. Sync strategy uses dynamodb.py to query/scan table or stream
5. Raw AWS responses flow into deserialize.py
6. Deserialized rows go into Singer message builder
7. Messages emitted to stdout
c. Architectural patterns & principles
• Plugin strategy pattern: sync_strategies folder holds interchangeable sync algorithms.
• Single Responsibility: each module has one focused job (discover vs sync vs deserialize).
• Decoupling: AWS interactions are isolated in a wrapper module.
• Testable design: modules can be unit-tested in isolation.
d. Key technologies & libraries
• Python 3 CLI
• boto3 (AWS SDK)
• Singer specification (JSON message schemas)
• pytest for testing
Final advice
• Sync_strategies under a “Strategy” container.
• Don’t forget the discovery path (discover.py → schema messages).
• Config & state file as inputs to the system.
• Aim for high detail: separate deserializer from output formatter, highlight AWS wrapper, and show the pluggable strategies clearly.