How chankein/teradatamigrate2redshift works
Project type and purpose
• This is a single‐script, command‐line DDL conversion tool written in Python.
• Purpose: read a Teradata DDL file, apply a set of type‐ and index‐conversion rules, then emit an equivalent Redshift DDL file plus conversion statistics.
File structure
• Top‐level:
• README.md – project description, rules, usage
• ddl_convertor.py – the entire implementation
• No subdirectories or external modules – all logic lives in ddl_convertor.py.
• Depends on Python 3.x and standard libraries (argparse, file I/O, possibly regex).
Key architectural insights from README
• Command‐line interface with three arguments: input file, output file, row count.
• Conversion rules fall into four domains: column‐type mapping, naming tweaks, distribution/style logic, sort/distribution key selection.
• Outputs both DDL and console statistics.
Main components
a. CLI Front End
• Parses “-i”, “-o”, “-r” args
• Validates inputs
b. DDL Parser & AST Builder
• Reads Teradata DDL text
• Tokenizes statements (table definitions, column specs, indexes)
• Builds an internal representation (e.g., Table objects, Column objects)
c. Conversion Engine
• Column Type Mapper – applies type conversion table
• Name Rule Applier – handles _UTC → BIGINT, _FLG → _flag
• Distribution & Sort Key Selector – uses PRIMARY INDEX/PARTITION INDEX + row count logic
d. DDL Formatter & Writer
• Translates internal objects to Redshift DDL strings
• Writes to the output file
e. Statistics Reporter
• Gathers counts of changed, recommended, and prohibited columns
• Prints summary to console
f. File System & Python Runtime
• External: file system reads/writes, Python 3 interpreter
Relationships & data flow
• CLI Front End → DDL Parser: passes input path
• DDL Parser → Conversion Engine: passes AST
• Conversion Engine → DDL Formatter: passes converted AST
• DDL Formatter → File System: writes output
• Conversion Engine & Formatter → Statistics Reporter → Console
Architectural patterns and principles
• Single‐pass pipeline: parse → transform → emit
• Separation of concerns within the script (even if in one file):
• Input handling
• Parsing
• Transformation
• Output & reporting
• Rule table‐driven conversion (type mappings defined in a lookup)
Diagramming recommendations
Input file → Parser → Converter → Formatter → Output file
Converter → Reporter → Console.
• Blue = Input/Output
• Green = Processing stages
• Orange = Reporting
• Gray = External (Python runtime, file system)
• Rectangles for modules
• Parallelograms for CLI/Console
Essential details to capture
• Argument parsing as the entry point
• Internal AST (or data structure) carrying table/column metadata
• The four rule sets (type, naming, distribution keys, distribution style)
• Output writer and statistics reporter