How andivoe/article-cleaner-for-sketchengine works
This repo is a single-node Python application (not a distributed system) that functions as a document-processing tool / pipeline for extracting text from academic PDFs and cleaning/removing non-body content, then exporting cleaned .docx files (and also exposing CLI, GUI, and Python API entry points).
Key signals from the repo:
• Single main module: pdf_cleaner.py
• Support scripts: verify_setup.py, run_pdf_cleaner.bat
• Dependency manifests: requirements.txt, pdf_cleaner_requirements.txt
• User docs: START_HERE.md, PDF_CLEANER_USER_GUIDE.md, etc.
• External libraries called out: PyMuPDF (fitz) for extraction; python-docx for output.
File structure analysis (how the project is organized)
Top-level is flat—no src/, no packages—which implies:
• Monolithic script/module containing both:
• Interface layers (GUI/CLI/API)
• Core processing pipeline (extract → clean → write)
• Configuration constants (e.g., SECTION_KEYWORDS, METADATA_PATTERNS in pdf_cleaner.py)
• pdf_cleaner.py: the core application
• run_pdf_cleaner.bat: Windows launcher (primarily for GUI mode)
• verify_setup.py: environment verification / dependency check
README insights to incorporate (interfaces, pipeline stages, outputs)
The README explicitly describes three interfaces:
• GUI mode (python pdf_cleaner.py --gui or run_pdf_cleaner.bat)
• CLI mode (--input, --output, --folder)
• Python API (process_pdf, extract_text_from_pdf, clean_text, save_to_docx)
And a canonical pipeline:
Clean text via
• section keyword removal (SECTION_KEYWORDS)
• regex/pattern matching (METADATA_PATTERNS)
• header/footer/page number stripping
Entry Points / Interfaces
• GUI Interface (Tkinter or similar; actual toolkit likely inside pdf_cleaner.py)
• CLI Interface (argparse-style command parsing)
• Python API (importable functions: process_pdf, extract_text_from_pdf, clean_text, save_to_docx)
Orchestration Layer
• “Batch Processor / Job Orchestrator”
• loops over files
• manages output folder creation (cleaned_articles/)
• aggregates success/failure
• triggers logging
Core Processing Pipeline
• PDF Reader / Text Extractor
• Library: PyMuPDF (fitz)
• Input: PDF file path/bytes
• Output: raw extracted text (+ maybe extraction success flag)
• Cleaner / Filter Engine
• Section remover (driven by SECTION_KEYWORDS)
• Pattern/regex remover (driven by METADATA_PATTERNS)
• Structural cleanup (headers/footers/page numbers)
• Output: cleaned body text
• Document Writer
• Library: python-docx
• Output: .docx file(s), preserving some formatting (as claimed)
Cross-Cutting Concerns
• Logging & Error Handling
• Extraction failures, bad PDFs, missing deps, etc.
• Configuration
• SECTION_KEYWORDS
• METADATA_PATTERNS
• runtime flags (CLI args / GUI options)
External Entities
• File System
• Input PDFs (user-selected path(s))
• Output directory cleaned_articles/
• External Libraries
• PyMuPDF (fitz)
• python-docx
Setup / Tooling (optional but useful)
• verify_setup.py (dependency/env verification)
• run_pdf_cleaner.bat (Windows launcher)
• GUI/CLI/API → Orchestrator
• Orchestrator → PDF Extractor (per file)
• PDF Extractor → Cleaner (raw text)
• Cleaner → DOCX Writer (cleaned text)
• DOCX Writer → File System (write .docx)
• Orchestrator → Logging (status/errors)
• Orchestrator → File System (create cleaned_articles/, enumerate folder inputs)
Also show:
• Cleaner ↔ Configuration (SECTION_KEYWORDS, METADATA_PATTERNS)
• Extractor → PyMuPDF (dependency)
• Writer → python-docx (dependency)
Architectural patterns used
This project is best represented as:
• Pipeline / ETL-style batch processor (Extract → Transform → Load/write)
• Ports & adapters (lightweight) via multiple entry points (GUI/CLI/API) feeding the same core pipeline
• Configuration-driven filtering (keywords + patterns)
You don’t need to force MVC; it’s closer to “script with layers”.
Relevant technologies
• Python 3.7+
• PyMuPDF (fitz) for extraction
• python-docx for output
• OS filesystem for input/output
“Sketch Engine target” as an external downstream consumer (not integrated, but a primary use case).
User runs tool → tool reads PDFs from filesystem → outputs DOCX to filesystem → optional downstream use in Sketch Engine.
GUI/CLI/API entry points calling the shared core pipeline within pdf_cleaner.py, plus external libs and filesystem.
Loop over PDFs:
• select files → for each file: extract → clean → write → log result.
This combination captures both *architecture* and the *pipeline mechanics* that matter for correctness.