# tom1030507/newspaper_job_extractor architecture

> Architecture diagram of the GitHub repository tom1030507/newspaper_job_extractor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tom1030507/newspaper_job_extractor
- Repository: https://github.com/tom1030507/newspaper_job_extractor

## Overview

##### Project type and purpose
- This is a standalone command-line image/PDF processing tool (“Newspaper Job Block Extraction Tool”) written in Python.
- Its goal is to take scanned newspaper pages (images or PDFs), detect job advertisement regions, and output each region as a separate image for downstream OCR or archiving.

##### File structure
- Top-level scripts and docs:
- main.py – entry point that loads files, invokes processing pipeline, writes outputs
- README.md / README.zh.md – project description and usage
- LICENSE – project licence
- newspaper/ – sample input PDFs and scanned PDFs
- job_section_success_rate_v1.xlsx – example analytics or reporting artifact (not core to extraction pipeline)

##### Key architectural insights from the README
- Uses OpenCV (cv2) and NumPy for image processing workflows
- Uses PyMuPDF (fitz) for PDF page rendering into images at configurable DPI
- Modular pipeline with these stages: preprocessing → contour detection → smart filtering → containment analysis → missing-area detection → final filtering → output generation
- Parameter configuration in main.py (min_dim, max_aspect_ratio, dpi, debug flag)

##### Main components
a. Input Loader
- PDF Renderer (PyMuPDF)
- Image Reader (OpenCV imread)
b. Preprocessing Module
- Grayscale conversion
- Gaussian blur
- Adaptive threshold
- Canny edge detection
c. Contour Detector
- FindContours on the edge map
d. Smart Filter
- Remove too large / too small / extreme aspect ratio regions
e. Containment Analyzer
- Hierarchical containment checks to avoid duplicates
f. Missing-Area Detector
- Create mask of detected blocks
- Morphological closing/opening
- Find new candidate regions from unfilled areas
g. Final Filter
- Re-apply containment checks across all candidates
h. Output Generator
- Save individual block images, debug images if enabled
i. Configuration & CLI
- Parameter definitions in main.py (min_dim, dpi, debug flag)
- Command line invocation: `python main.py`

##### Relationships and data flow
- Data flow:
1. Input Loader → Preprocessing Module
2. Preprocessed image → Contour Detector
3. Contours → Smart Filter → Containment Analyzer → Missing-Area Detector → Final Filter
4. Final blocks → Output Generator → File System (output folder)
- All modules depend on external libraries: OpenCV, NumPy, PyMuPDF
- Config parameters flow into each stage as needed

##### Architectural patterns and design principles
- Pipeline / chain of responsibility: each processing stage accepts an image or candidate list and outputs refined results
- Single responsibility: each module has a clear, isolated task
- Configurability via global parameters (min_dim, max_aspect_ratio, dpi, debug)
- Extensibility point: easy to insert new filtering stages or swap out detection algorithms

##### Elements
- Rectangles for each module (Input Loader, Preprocessor, Contour Detector, etc.)
- Blue for core algorithm modules
- Green for I/O modules (Input Loader, Output Generator)
- Orange for external dependencies

##### Advice for high-fidelity detail
- Every processing stage separately – don’t lump filtering and containment analysis together
- Configuration parameters flowing into relevant modules
- If space permits, annotate primary algorithms (e.g., “adaptive threshold (blockSize=11, C=2)”)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Entry
    CLI["python main.py"]:::io

    %% File System Input
    FSInput[(Input PDFs/Images)]:::io

    %% Modules
    CLI --> FSInput
    FSInput --> InputLoader["Input Loader\n(PDF Renderer, Image Reader)"]:::io
    click InputLoader "https://github.com/tom1030507/newspaper_job_extractor/blob/main/main.py"

    InputLoader --> Preprocessing["Preprocessing Module\n(grayscale, blur, threshold, Canny)"]:::core
    click Preprocessing "https://github.com/tom1030507/newspaper_job_extractor/blob/main/main.py"

    Preprocessing --> Contour["Contour Detector\n(FindContours)"]:::core
    click Contour "https://github.com/tom1030507/newspaper_job_extractor/blob/main/main.py"

    Contour --> SmartFilter["Smart Filter\n(size & aspect ratio)"]:::core
    click SmartFilter "https://github.com/tom1030507/newspaper_job_extractor/blob/main/main.py"

    SmartFilter --> Containment["Containment Analyzer\n(hierarchical checks)"]:::core
    click Containment "https://github.com/tom1030507/newspaper_job_extractor/blob/main/main.py"

    Containment --> MissingArea["Missing-Area Detector\n(mask & morphology)"]:::core
    click MissingArea "https://github.com/tom1030507/newspaper_job_extractor/blob/main/main.py"

    MissingArea --> FinalFilter["Final Filter\n(reapply filters)"]:::core
    click FinalFilter "https://github.com/tom1030507/newspaper_job_extractor/blob/main/main.py"

    FinalFilter --> OutputGen["Output Generator\n(save blocks & debug outputs)"]:::io
    click OutputGen "https://github.com/tom1030507/newspaper_job_extractor/blob/main/main.py"

    OutputGen --> Output[(Extracted Blocks)]:::io

    %% External Dependencies
    cv2[(cv2)]:::ext
    NumPy[(NumPy)]:::ext
    PyMuPDF[(PyMuPDF)]:::ext

    PyMuPDF --> InputLoader

    cv2 --> Preprocessing
    cv2 --> Contour
    cv2 --> SmartFilter
    cv2 --> Containment
    cv2 --> MissingArea
    cv2 --> FinalFilter

    NumPy --> Preprocessing
    NumPy --> Contour
    NumPy --> SmartFilter
    NumPy --> Containment
    NumPy --> MissingArea
    NumPy --> FinalFilter

    %% Configuration parameters
    Config["Config & CLI\n(min_dim, max_aspect_ratio, dpi, debug)"]:::io
    Config --> InputLoader
    Config --> Preprocessing
    Config --> SmartFilter
    Config --> Containment
    Config --> MissingArea
    Config --> FinalFilter
    click Config "https://github.com/tom1030507/newspaper_job_extractor/blob/main/main.py"

    %% Click event for input files
    click FSInput "https://github.com/tom1030507/newspaper_job_extractor/blob/main/newspaper/*.pdf"

    %% Styles
    classDef core fill:#D2E5FF,stroke:#1E90FF;
    classDef io fill:#E6F8E0,stroke:#228B22;
    classDef ext fill:#FFE5B4,stroke:#FF8C00;
```
