# andivoe/article-cleaner-for-sketchengine architecture

> Architecture diagram of the GitHub repository andivoe/article-cleaner-for-sketchengine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/andivoe/article-cleaner-for-sketchengine
- Repository: https://github.com/andivoe/article-cleaner-for-sketchengine

## Overview

This repo is a **single-node Python application** (not a distributed system) that functions as a **document-processing tool / pipeline** for **extracting text from academic PDFs and cleaning/removing non-body content**, then exporting **cleaned .docx** files (and also exposing **CLI, GUI, and Python API** entry points).

Key signals from the repo:
- Single main module: `pdf_cleaner.py`
- Support scripts: `verify_setup.py`, `run_pdf_cleaner.bat`
- Dependency manifests: `requirements.txt`, `pdf_cleaner_requirements.txt`
- User docs: `START_HERE.md`, `PDF_CLEANER_USER_GUIDE.md`, etc.
- External libraries called out: **PyMuPDF (fitz)** for extraction; **python-docx** for output.

##### File structure analysis (how the project is organized)
Top-level is flat—no `src/`, no packages—which implies:
- **Monolithic script/module** containing both:
- Interface layers (GUI/CLI/API)
- Core processing pipeline (extract → clean → write)
- Configuration constants (e.g., `SECTION_KEYWORDS`, `METADATA_PATTERNS` in `pdf_cleaner.py`)

- `pdf_cleaner.py`: the core application
- `run_pdf_cleaner.bat`: Windows launcher (primarily for GUI mode)
- `verify_setup.py`: environment verification / dependency check

##### README insights to incorporate (interfaces, pipeline stages, outputs)
The README explicitly describes **three interfaces**:
- GUI mode (`python pdf_cleaner.py --gui` or `run_pdf_cleaner.bat`)
- CLI mode (`--input`, `--output`, `--folder`)
- Python API (`process_pdf`, `extract_text_from_pdf`, `clean_text`, `save_to_docx`)

And a **canonical pipeline**:
##### Clean text via
- section keyword removal (`SECTION_KEYWORDS`)
- regex/pattern matching (`METADATA_PATTERNS`)
- header/footer/page number stripping

##### Entry Points / Interfaces
- **GUI Interface** (Tkinter or similar; actual toolkit likely inside `pdf_cleaner.py`)
- **CLI Interface** (argparse-style command parsing)
- **Python API** (importable functions: `process_pdf`, `extract_text_from_pdf`, `clean_text`, `save_to_docx`)

##### Orchestration Layer
- “Batch Processor / Job Orchestrator”
- loops over files
- manages output folder creation (`cleaned_articles/`)
- aggregates success/failure
- triggers logging

##### Core Processing Pipeline
- **PDF Reader / Text Extractor**
- Library: PyMuPDF (`fitz`)
- Input: PDF file path/bytes
- Output: raw extracted text (+ maybe extraction success flag)
- **Cleaner / Filter Engine**
- Section remover (driven by `SECTION_KEYWORDS`)
- Pattern/regex remover (driven by `METADATA_PATTERNS`)
- Structural cleanup (headers/footers/page numbers)
- Output: cleaned body text
- **Document Writer**
- Library: python-docx
- Output: `.docx` file(s), preserving some formatting (as claimed)

##### Cross-Cutting Concerns
- **Logging & Error Handling**
- Extraction failures, bad PDFs, missing deps, etc.
- **Configuration**
- `SECTION_KEYWORDS`
- `METADATA_PATTERNS`
- runtime flags (CLI args / GUI options)

##### External Entities
- **File System**
- Input PDFs (user-selected path(s))
- Output directory `cleaned_articles/`
- **External Libraries**
- PyMuPDF (fitz)
- python-docx

##### Setup / Tooling (optional but useful)
- `verify_setup.py` (dependency/env verification)
- `run_pdf_cleaner.bat` (Windows launcher)

- GUI/CLI/API → Orchestrator
- Orchestrator → PDF Extractor (per file)
- PDF Extractor → Cleaner (raw text)
- Cleaner → DOCX Writer (cleaned text)
- DOCX Writer → File System (write .docx)
- Orchestrator → Logging (status/errors)
- Orchestrator → File System (create `cleaned_articles/`, enumerate folder inputs)

Also show:
- Cleaner ↔ Configuration (SECTION_KEYWORDS, METADATA_PATTERNS)
- Extractor → PyMuPDF (dependency)
- Writer → python-docx (dependency)

##### Architectural patterns used
This project is best represented as:
- **Pipeline / ETL-style batch processor** (Extract → Transform → Load/write)
- **Ports & adapters (lightweight)** via multiple entry points (GUI/CLI/API) feeding the same core pipeline
- **Configuration-driven filtering** (keywords + patterns)

You don’t need to force MVC; it’s closer to “script with layers”.

##### Relevant technologies
- Python 3.7+
- PyMuPDF (fitz) for extraction
- python-docx for output
- OS filesystem for input/output
“Sketch Engine target” as an external downstream consumer (not integrated, but a primary use case).

User runs tool → tool reads PDFs from filesystem → outputs DOCX to filesystem → optional downstream use in Sketch Engine.

GUI/CLI/API entry points calling the shared core pipeline within `pdf_cleaner.py`, plus external libs and filesystem.

Loop over PDFs:
- select files → for each file: extract → clean → write → log result.

This combination captures both *architecture* and the *pipeline mechanics* that matter for correctness.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% ========= External Actors & OS =========
    User(["User (Researcher)"]):::actor
    OS["Operating System (Local Machine)"]:::external

    FS_IN[("File System\nInput PDFs")]:::store
    FS_OUT[("File System\nOutput: cleaned_articles/")]:::store
    Sketch["Sketch Engine\n(Downstream consumer)"]:::external

    %% ========= Tooling / Setup (non-runtime) =========
    subgraph "Setup & Tooling (Non-runtime)"
        direction TB
        Launcher["Windows Launcher\nrun_pdf_cleaner.bat"]:::tool
        Verify["Environment Verifier\nverify_setup.py"]:::tool
        Req["requirements.txt"]:::manifest
        ReqAlt["pdf_cleaner_requirements.txt"]:::manifest
        Docs["Documentation\nREADME / Guides / Notes"]:::docs
    end

    %% ========= Application (Single Process) =========
    subgraph "Application (Single-node Python Process)"
        direction TB

        subgraph "pdf_cleaner.py (Single Deployable Unit)"
            direction TB

            %% Entry points / Interfaces
            subgraph "Entry Points / Interfaces"
                direction TB
                GUI(["GUI Mode\n(--gui)"]):::ui
                CLI(["CLI Mode\n(--input/--output/--folder)"]):::ui
                API(["Python API\nprocess_pdf()"]):::ui
            end

            %% Orchestration
            Orchestrator["Batch Processor / Job Orchestrator\n(loop files, create output dir)"]:::core

            %% Pipeline
            Extractor["PDF Reader / Text Extractor\nextract_text_from_pdf()"]:::stage
            Cleaner["Cleaner / Filter Engine\nclean_text()"]:::stage
            Writer["Document Writer\nsave_to_docx()"]:::stage

            %% Cross-cutting
            Config["Configuration Constants\nSECTION_KEYWORDS\nMETADATA_PATTERNS"]:::config
            Log["Logging & Error Handling"]:::crosscut
        end
    end

    %% ========= External Libraries =========
    subgraph "External Libraries (Dependencies)"
        direction TB
        PyMuPDF{{"PyMuPDF (fitz)"}}:::dep
        PyDocx{{"python-docx"}}:::dep
        PyRuntime{{"Python 3.7+"}}:::dep
    end

    %% ========= Context / Invocation Paths =========
    User -->|"runs"| OS
    OS -->|"launches"| Launcher
    User -->|"starts"| GUI
    User -->|"invokes"| CLI
    User -->|"imports_calls"| API

    Launcher -->|"starts_gui_mode"| GUI

    %% ========= Interface -> Core =========
    GUI -->|"selected_PDFs_or_folder"| Orchestrator
    CLI -->|"args(PDF_paths,folder,output)"| Orchestrator
    API -->|"PDF_path(s)"| Orchestrator

    %% ========= Orchestration + FS =========
    Orchestrator -->|"enumerate_inputs"| FS_IN
    Orchestrator -->|"create_or_use_output_dir"| FS_OUT

    %% ========= Pipeline Dataflow =========
    Orchestrator -->|"for_each_PDF_path"| Extractor
    Extractor -->|"read_PDF_bytes"| FS_IN
    Extractor -->|"raw_extracted_text"| Cleaner
    Cleaner -->|"cleaned_body_text"| Writer
    Writer -->|".docx_file"| FS_OUT

    %% ========= Cross-cutting dependencies =========
    Orchestrator -->|"status_errors"| Log
    Extractor -->|"exceptions"| Log
    Cleaner -->|"warnings"| Log
    Writer -->|"write_errors"| Log

    Config -.->|"keywords_patterns"| Cleaner

    %% ========= Library bindings =========
    Extractor -->|"uses"| PyMuPDF
    Writer -->|"uses"| PyDocx
    GUI -->|"runs_on"| PyRuntime
    CLI -->|"runs_on"| PyRuntime
    API -->|"runs_on"| PyRuntime

    %% ========= Downstream (manual) =========
    FS_OUT -.->|"manual_upload_use"| Sketch

    %% ========= Tooling relationships =========
    Verify -.->|"checks_deps"| Req
    Verify -.->|"checks_deps"| ReqAlt
    Verify -.->|"validates_runtime"| PyRuntime
    Docs -.->|"explains_usage"| GUI
    Docs -.->|"explains_usage"| CLI
    Docs -.->|"explains_usage"| API
    Req -.->|"lists"| PyMuPDF
    Req -.->|"lists"| PyDocx
    ReqAlt -.->|"pins_alt"| PyMuPDF
    ReqAlt -.->|"pins_alt"| PyDocx

    %% ========= Click Events (from component mapping) =========
    click GUI "https://github.com/andivoe/article-cleaner-for-sketchengine/blob/main/pdf_cleaner.py"
    click CLI "https://github.com/andivoe/article-cleaner-for-sketchengine/blob/main/pdf_cleaner.py"
    click API "https://github.com/andivoe/article-cleaner-for-sketchengine/blob/main/pdf_cleaner.py"
    click Orchestrator "https://github.com/andivoe/article-cleaner-for-sketchengine/blob/main/pdf_cleaner.py"
    click Extractor "https://github.com/andivoe/article-cleaner-for-sketchengine/blob/main/pdf_cleaner.py"
    click Cleaner "https://github.com/andivoe/article-cleaner-for-sketchengine/blob/main/pdf_cleaner.py"
    click Writer "https://github.com/andivoe/article-cleaner-for-sketchengine/blob/main/pdf_cleaner.py"
    click Config "https://github.com/andivoe/article-cleaner-for-sketchengine/blob/main/pdf_cleaner.py"
    click Log "https://github.com/andivoe/article-cleaner-for-sketchengine/blob/main/pdf_cleaner.py"

    click Launcher "https://github.com/andivoe/article-cleaner-for-sketchengine/blob/main/run_pdf_cleaner.bat"
    click Verify "https://github.com/andivoe/article-cleaner-for-sketchengine/blob/main/verify_setup.py"
    click Req "https://github.com/andivoe/article-cleaner-for-sketchengine/blob/main/requirements.txt"
    click ReqAlt "https://github.com/andivoe/article-cleaner-for-sketchengine/blob/main/pdf_cleaner_requirements.txt"

    click Docs "https://github.com/andivoe/article-cleaner-for-sketchengine/blob/main/README.md"

    %% ========= Styles =========
    classDef actor fill:#0b3d91,stroke:#062a62,color:#ffffff,stroke-width:1px
    classDef external fill:#f3f4f6,stroke:#9ca3af,color:#111827,stroke-width:1px
    classDef ui fill:#dbeafe,stroke:#2563eb,color:#0f172a,stroke-width:1px
    classDef core fill:#e0f2fe,stroke:#0284c7,color:#0f172a,stroke-width:1px
    classDef stage fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:1px
    classDef crosscut fill:#fef3c7,stroke:#d97706,color:#111827,stroke-width:1px
    classDef config fill:#fae8ff,stroke:#a21caf,color:#111827,stroke-width:1px
    classDef store fill:#fff7ed,stroke:#ea580c,color:#111827,stroke-width:1px
    classDef dep fill:#ede9fe,stroke:#6d28d9,color:#111827,stroke-width:1px
    classDef tool fill:#f1f5f9,stroke:#334155,color:#0f172a,stroke-width:1px,stroke-dasharray: 4 3
    classDef manifest fill:#f1f5f9,stroke:#334155,color:#0f172a,stroke-width:1px,stroke-dasharray: 4 3
    classDef docs fill:#f1f5f9,stroke:#334155,color:#0f172a,stroke-width:1px,stroke-dasharray: 4 3
```
