How tom1030507/newspaper_job_extractor works
Project type and purpose
• This is a standalone command-line image/PDF processing tool (“Newspaper Job Block Extraction Tool”) written in Python.
• Its goal is to take scanned newspaper pages (images or PDFs), detect job advertisement regions, and output each region as a separate image for downstream OCR or archiving.
File structure
• Top-level scripts and docs:
• main.py – entry point that loads files, invokes processing pipeline, writes outputs
• README.md / README.zh.md – project description and usage
• LICENSE – project licence
• newspaper/ – sample input PDFs and scanned PDFs
• job_section_success_rate_v1.xlsx – example analytics or reporting artifact (not core to extraction pipeline)
Key architectural insights from the README
• Uses OpenCV (cv2) and NumPy for image processing workflows
• Uses PyMuPDF (fitz) for PDF page rendering into images at configurable DPI
• Modular pipeline with these stages: preprocessing → contour detection → smart filtering → containment analysis → missing-area detection → final filtering → output generation
• Parameter configuration in main.py (min_dim, max_aspect_ratio, dpi, debug flag)
Main components
a. Input Loader
• PDF Renderer (PyMuPDF)
• Image Reader (OpenCV imread)
b. Preprocessing Module
• Grayscale conversion
• Gaussian blur
• Adaptive threshold
• Canny edge detection
c. Contour Detector
• FindContours on the edge map
d. Smart Filter
• Remove too large / too small / extreme aspect ratio regions
e. Containment Analyzer
• Hierarchical containment checks to avoid duplicates
f. Missing-Area Detector
• Create mask of detected blocks
• Morphological closing/opening
• Find new candidate regions from unfilled areas
g. Final Filter
• Re-apply containment checks across all candidates
h. Output Generator
• Save individual block images, debug images if enabled
i. Configuration & CLI
• Parameter definitions in main.py (min_dim, dpi, debug flag)
• Command line invocation: python main.py
Relationships and data flow
• Data flow:
1. Input Loader → Preprocessing Module
2. Preprocessed image → Contour Detector
3. Contours → Smart Filter → Containment Analyzer → Missing-Area Detector → Final Filter
4. Final blocks → Output Generator → File System (output folder)
• All modules depend on external libraries: OpenCV, NumPy, PyMuPDF
• Config parameters flow into each stage as needed
Architectural patterns and design principles
• Pipeline / chain of responsibility: each processing stage accepts an image or candidate list and outputs refined results
• Single responsibility: each module has a clear, isolated task
• Configurability via global parameters (min_dim, max_aspect_ratio, dpi, debug)
• Extensibility point: easy to insert new filtering stages or swap out detection algorithms
Elements
• Rectangles for each module (Input Loader, Preprocessor, Contour Detector, etc.)
• Blue for core algorithm modules
• Green for I/O modules (Input Loader, Output Generator)
• Orange for external dependencies
Advice for high-fidelity detail
• Every processing stage separately – don’t lump filtering and containment analysis together
• Configuration parameters flowing into relevant modules
• If space permits, annotate primary algorithms (e.g., “adaptive threshold (blockSize=11, C=2)”)