How patrykbala/documentner works
Project type and purpose
• This is a research‐oriented, full‐pipeline data‐processing application for document image OCR + Named Entity Recognition (NER).
• Purpose: Take scanned or photographed business cards, extract raw text via OCR, clean and preprocess it, label entities (BIO), train an NER model with spaCy, then run predictions to output named units.
File structure
• Top‐level notebooks (01_Pytesseract.ipynb … 05_Final_predictions.ipynb) map the evolution of pipeline steps: OCR → data prep → preprocessing → predictions → final results.
• Key scripts:
• predictions.py houses the end‐to‐end inference pipeline (image → entities).
• preprocess.py handles text cleaning, tokenization, BIO labeling.
• Data directory: data/ holds training/test pickles and spaCy binary files.
• Output directory: output/model‐* contains spaCy model artifacts (config, vocab, tokenizer, trained NER weights).
• Config files (base_config.cfg, config.cfg) store hyperparameters and pipeline definitions.
README for additional insights
• Clarifies pipeline stages, used technologies (OpenCV, pytesseract for OCR; pandas, regex, spaCy for NER).
a. Main components
• Document Ingestion (Desktop/Mobile Upload)
• OCR Subsystem
– OpenCV image preprocessing
– Tesseract OCR engine
• Text Cleaning & Preprocessing
– RegEx cleaning
– Tokenization
– BIO labeling (for training)
• Model Training
– spaCy pipeline: tok2vec → NER → optimizer
– Training data (BIO‐labeled)
– Config management
• Inference/Prediction
– predictions.py loads trained spaCy model
– Applies OCR + preprocessing + NER on new documents
• Storage & Artifacts
– Raw data (data/)
– Pickled data (*.pickle)
– spaCy models & metadata (output/model‐*)
b. Relationships & data flow
– Branch: Preprocessing → (Labeling → Training) and Preprocessing → (Inference)
– Model Training writes to Model Artifacts storage
– Inference reads Model Artifacts, outputs Named Entities
c. Architectural patterns / design principles
– Pipeline architecture: clear, ordered stages
– Modular design: separate scripts/notebooks for each stage
– Configuration‐driven: config.cfg controls hyperparameters
– Reusable components: preprocessing functions shared between training & inference
d. Key technologies/frameworks
– Python, Jupyter Notebooks
– OpenCV, pytesseract
– pandas, regex for text processing
– spaCy for NER training & inference
– File‐based config (cfg)