# drexeltriangle/wordpress-etl architecture

> Architecture diagram of the GitHub repository drexeltriangle/wordpress-etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/drexeltriangle/wordpress-etl
- Repository: https://github.com/drexeltriangle/wordpress-etl

## Overview

##### Project Type and Purpose
- This is a command-line ETL (Extract-Transform-Load) tool written in Python. Its purpose is to migrate WordPress export data (XML inside a ZIP) into a local MariaDB instance (PRISM), performing extraction, translation, sanitization, and SQL formatting/injection.

##### File Structure Analysis
Top-level files and directories:
- main.py / App.py – CLI entrypoints that orchestrate the pipeline
- Animator.py – (likely optional visualization or progress reporting)
- Data/wp-export.zip – the raw WordPress XML export
- Extractor.py – unzips and parses XML → Python dict
- Translator/… – maps raw dicts → domain objects (Authors, GuestAuthors, Articles)
- Sanitizer/… – applies Policy classes to clean and normalize object fields
- Formatter/… – generates SQL INSERT statements for MariaDB
- Utils/… – shared constants, NLP helpers, generic utility functions
- requirements.txt – Python dependencies

##### Key Components
a. Data Source
– WordPress Export (Data/wp-export.zip containing posts.xml, guest_authors.xml)
b. Extractor Module
– Extractor.py
– Output: raw Python dicts under keys “auth”, “guestAuth”, “art”
c. Translator Subsystem
– Translator.py + child classes (ArticleTranslator, AuthorTranslator, GuestAuthorTranslator)
– Transforms raw dict → strongly-typed objects in dictionaries
d. Sanitizer Subsystem
– Sanitizer.py + Policy classes (AuthorPolicy, GuestAuthorPolicy, DiffChecker, etc.)
– Cleans edge cases: XML escapes, aggregated users, name splitting rules
e. Formatter Subsystem
– Formatter.py + formatters (ArticleFormatter, AuthorFormatter, GuestAuthorFormatter)
– Emits SQL text files or optionally injects directly to PRISM DB
f. Injector / PRISM Connector (future)
– CLI logic in App.py or main.py to open DB connection, execute SQL
g. Support Utilities
– Utils for constants and NLP-based sanitization

##### Relationships & Data Flow
– WordPress ZIP → Extractor
– Extractor ⇒ raw dicts → Translator
– Translator ⇒ domain objects → Sanitizer
– Sanitizer ⇒ cleaned objects → Formatter
– Formatter ⇒ SQL script or DB injector → PRISM (MariaDB)
– Optional profiling with Scalene

##### Architectural Patterns & Principles
– Pipeline / staged ETL architecture
– Single Responsibility per module: each folder handles one stage
– Policy‐based sanitization for pluggable data‐cleaning rules
– Translator inheritance for extensibility (open/closed principle)
– Separation of concerns: raw I/O, transformation logic, sanitization, and SQL generation

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Source
    subgraph "Data Source"
        DS["WordPress Export ZIP"]:::data
    end

    %% Extractor Stage
    subgraph "Extractor Module"
        EX["Extractor.py"]:::extractor
    end

    %% Translator Stage
    subgraph "Translator Subsystem"
        TBase["Translator/Translator.py"]:::translator
        TArt["Translator/ArticleTranslator.py"]:::translator
        TAuth["Translator/AuthorTranslator.py"]:::translator
        TGuest["Translator/GuestAuthorTranslator.py"]:::translator
        ObjArt["Translator/Article.py"]:::translator
        ObjAuth["Translator/Author.py"]:::translator
        ObjGuest["Translator/GuestAuthor.py"]:::translator
        ObjWP["Translator/WPObject.py"]:::translator
    end

    %% Sanitizer Stage
    subgraph "Sanitizer Subsystem"
        SBase["Sanitizer/Sanitizer.py"]:::sanitizer
        SSAuthor["Sanitizer/AuthorSanitizer.py"]:::sanitizer
        Pol["Sanitizer/Policy.py"]:::sanitizer
        PolAuth["Sanitizer/AuthorPolicy.py"]:::sanitizer
        PolGuest["Sanitizer/GuestAuthorPolicy.py"]:::sanitizer
        Diff["Sanitizer/DiffChecker.py"]:::sanitizer
    end

    %% Formatter Stage
    subgraph "Formatter Subsystem"
        FBase["Formatter/Formatter.py"]:::formatter
        FArt["Formatter/ArticleFormatter.py"]:::formatter
        FAuth["Formatter/AuthorFormatter.py"]:::formatter
        FGuest["Formatter/GuestAuthorFormatter.py"]:::formatter
    end

    %% Injector / Orchestrator
    subgraph "Orchestration & DB Injection"
        Main["main.py"]:::injector
        AppPy["App.py"]:::injector
    end

    %% Output Targets
    SQLDoc["SQL Script"]:::document
    DB["PRISM MariaDB"]:::database

    %% Support Utilities
    subgraph "Support Utilities"
        Cnst["Utils/Constants.py"]:::util
        NLP["Utils/NLP.py"]:::util
        UtilGen["Utils/Utility.py"]:::util
    end

    %% Optional Profiling/Visualization
    Anim["Animator.py"]:::optional
    Req["requirements.txt"]:::optional

    %% Data Flow
    DS -->|"ZIP/XML → dict"| EX
    EX -->|"dict"| TBase
    TBase -->|"dict → objects"| SSAuthor
    TArt -->|"objects"| SSAuthor
    TAuth -->|"objects"| SSAuthor
    TGuest -->|"objects"| SSAuthor
    SSAuthor -->|"objects → cleaned objects"| SBase
    SBase -->|"cleaned objects"| FBase
    FBase -->|"cleaned objects → SQL"| SQLDoc
    FBase -->|"direct inject"| Main
    Main -->|"executes SQL"| DB
    AppPy --> Main

    %% Utils linkage
    Cnst --> EX
    Cnst --> TBase
    Cnst --> SBase
    Cnst --> FBase
    NLP --> SBase
    UtilGen --> EX
    UtilGen --> TBase
    UtilGen --> SBase
    UtilGen --> FBase

    %% Optional
    Anim --> EX
    Req -->|"profiling"| EX

    %% Click Events
    click DS "https://github.com/drexeltriangle/wordpress-etl/blob/main/Data/wp-export.zip"
    click EX "https://github.com/drexeltriangle/wordpress-etl/blob/main/Extractor.py"
    click TBase "https://github.com/drexeltriangle/wordpress-etl/blob/main/Translator/Translator.py"
    click TArt "https://github.com/drexeltriangle/wordpress-etl/blob/main/Translator/ArticleTranslator.py"
    click TAuth "https://github.com/drexeltriangle/wordpress-etl/blob/main/Translator/AuthorTranslator.py"
    click TGuest "https://github.com/drexeltriangle/wordpress-etl/blob/main/Translator/GuestAuthorTranslator.py"
    click ObjArt "https://github.com/drexeltriangle/wordpress-etl/blob/main/Translator/Article.py"
    click ObjAuth "https://github.com/drexeltriangle/wordpress-etl/blob/main/Translator/Author.py"
    click ObjGuest "https://github.com/drexeltriangle/wordpress-etl/blob/main/Translator/GuestAuthor.py"
    click ObjWP "https://github.com/drexeltriangle/wordpress-etl/blob/main/Translator/WPObject.py"
    click SBase "https://github.com/drexeltriangle/wordpress-etl/blob/main/Sanitizer/Sanitizer.py"
    click SSAuthor "https://github.com/drexeltriangle/wordpress-etl/blob/main/Sanitizer/AuthorSanitizer.py"
    click Pol "https://github.com/drexeltriangle/wordpress-etl/blob/main/Sanitizer/Policy.py"
    click PolAuth "https://github.com/drexeltriangle/wordpress-etl/blob/main/Sanitizer/AuthorPolicy.py"
    click PolGuest "https://github.com/drexeltriangle/wordpress-etl/blob/main/Sanitizer/GuestAuthorPolicy.py"
    click Diff "https://github.com/drexeltriangle/wordpress-etl/blob/main/Sanitizer/DiffChecker.py"
    click FBase "https://github.com/drexeltriangle/wordpress-etl/blob/main/Formatter/Formatter.py"
    click FArt "https://github.com/drexeltriangle/wordpress-etl/blob/main/Formatter/ArticleFormatter.py"
    click FAuth "https://github.com/drexeltriangle/wordpress-etl/blob/main/Formatter/AuthorFormatter.py"
    click FGuest "https://github.com/drexeltriangle/wordpress-etl/blob/main/Formatter/GuestAuthorFormatter.py"
    click Main "https://github.com/drexeltriangle/wordpress-etl/blob/main/main.py"
    click AppPy "https://github.com/drexeltriangle/wordpress-etl/blob/main/App.py"
    click Cnst "https://github.com/drexeltriangle/wordpress-etl/blob/main/Utils/Constants.py"
    click NLP "https://github.com/drexeltriangle/wordpress-etl/blob/main/Utils/NLP.py"
    click UtilGen "https://github.com/drexeltriangle/wordpress-etl/blob/main/Utils/Utility.py"
    click Anim "https://github.com/drexeltriangle/wordpress-etl/blob/main/Animator.py"
    click Req "https://github.com/drexeltriangle/wordpress-etl/blob/main/requirements.txt"

    %% Styles
    classDef data fill:#D6E9FF,stroke:#4D8FAC,color:#000
    classDef extractor fill:#CCE5FF,stroke:#3399FF,color:#000
    classDef translator fill:#DFF2BF,stroke:#8FBC8F,color:#000
    classDef sanitizer fill:#FFE5B4,stroke:#FFA500,color:#000
    classDef formatter fill:#FFCCCC,stroke:#FF6666,color:#000
    classDef injector fill:#E0BBE4,stroke:#957DAD,color:#000
    classDef database fill:#FFD700,stroke:#DAA520,color:#000
    classDef document fill:#F5F5DC,stroke:#CCCC99,color:#000
    classDef util fill:#E8E8E8,stroke:#B0B0B0,color:#000
    classDef optional fill:#F0F0F0,stroke:#999999,color:#666
```
