How drexeltriangle/wordpress-etl works
Project Type and Purpose
• This is a command-line ETL (Extract-Transform-Load) tool written in Python. Its purpose is to migrate WordPress export data (XML inside a ZIP) into a local MariaDB instance (PRISM), performing extraction, translation, sanitization, and SQL formatting/injection.
File Structure Analysis
Top-level files and directories:
• main.py / App.py – CLI entrypoints that orchestrate the pipeline
• Animator.py – (likely optional visualization or progress reporting)
• Data/wp-export.zip – the raw WordPress XML export
• Extractor.py – unzips and parses XML → Python dict
• Translator/… – maps raw dicts → domain objects (Authors, GuestAuthors, Articles)
• Sanitizer/… – applies Policy classes to clean and normalize object fields
• Formatter/… – generates SQL INSERT statements for MariaDB
• Utils/… – shared constants, NLP helpers, generic utility functions
• requirements.txt – Python dependencies
Key Components
a. Data Source
– WordPress Export (Data/wp-export.zip containing posts.xml, guest_authors.xml)
b. Extractor Module
– Extractor.py
– Output: raw Python dicts under keys “auth”, “guestAuth”, “art”
c. Translator Subsystem
– Translator.py + child classes (ArticleTranslator, AuthorTranslator, GuestAuthorTranslator)
– Transforms raw dict → strongly-typed objects in dictionaries
d. Sanitizer Subsystem
– Sanitizer.py + Policy classes (AuthorPolicy, GuestAuthorPolicy, DiffChecker, etc.)
– Cleans edge cases: XML escapes, aggregated users, name splitting rules
e. Formatter Subsystem
– Formatter.py + formatters (ArticleFormatter, AuthorFormatter, GuestAuthorFormatter)
– Emits SQL text files or optionally injects directly to PRISM DB
f. Injector / PRISM Connector (future)
– CLI logic in App.py or main.py to open DB connection, execute SQL
g. Support Utilities
– Utils for constants and NLP-based sanitization
Relationships & Data Flow
– WordPress ZIP → Extractor
– Extractor ⇒ raw dicts → Translator
– Translator ⇒ domain objects → Sanitizer
– Sanitizer ⇒ cleaned objects → Formatter
– Formatter ⇒ SQL script or DB injector → PRISM (MariaDB)
– Optional profiling with Scalene
Architectural Patterns & Principles
– Pipeline / staged ETL architecture
– Single Responsibility per module: each folder handles one stage
– Policy‐based sanitization for pluggable data‐cleaning rules
– Translator inheritance for extensibility (open/closed principle)
– Separation of concerns: raw I/O, transformation logic, sanitization, and SQL generation