# patrykbala/documentner architecture

> Architecture diagram of the GitHub repository patrykbala/documentner, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/patrykbala/documentner
- Repository: https://github.com/patrykbala/documentner

## Overview

##### Project type and purpose
- This is a research‐oriented, full‐pipeline data‐processing application for document image OCR + Named Entity Recognition (NER).
- Purpose: Take scanned or photographed business cards, extract raw text via OCR, clean and preprocess it, label entities (BIO), train an NER model with spaCy, then run predictions to output named units.

##### File structure
- Top‐level notebooks (`01_Pytesseract.ipynb` … `05_Final_predictions.ipynb`) map the evolution of pipeline steps: OCR → data prep → preprocessing → predictions → final results.
- Key scripts:
- `predictions.py` houses the end‐to‐end inference pipeline (image → entities).
- `preprocess.py` handles text cleaning, tokenization, BIO labeling.
- Data directory: `data/` holds training/test pickles and spaCy binary files.
- Output directory: `output/model‐*` contains spaCy model artifacts (config, vocab, tokenizer, trained NER weights).
- Config files (`base_config.cfg`, `config.cfg`) store hyperparameters and pipeline definitions.

##### README for additional insights
- Clarifies pipeline stages, used technologies (OpenCV, pytesseract for OCR; pandas, regex, spaCy for NER).

a. Main components
- Document Ingestion (Desktop/Mobile Upload)
- OCR Subsystem
– OpenCV image preprocessing
– Tesseract OCR engine
- Text Cleaning & Preprocessing
– RegEx cleaning
– Tokenization
– BIO labeling (for training)
- Model Training
– spaCy pipeline: tok2vec → NER → optimizer
– Training data (BIO‐labeled)
– Config management
- Inference/Prediction
– `predictions.py` loads trained spaCy model
– Applies OCR + preprocessing + NER on new documents
- Storage & Artifacts
– Raw data (`data/`)
– Pickled data (`*.pickle`)
– spaCy models & metadata (`output/model‐*`)

b. Relationships & data flow
– Branch: Preprocessing → (Labeling → Training) and Preprocessing → (Inference)
– Model Training writes to Model Artifacts storage
– Inference reads Model Artifacts, outputs Named Entities

c. Architectural patterns / design principles
– Pipeline architecture: clear, ordered stages
– Modular design: separate scripts/notebooks for each stage
– Configuration‐driven: `config.cfg` controls hyperparameters
– Reusable components: preprocessing functions shared between training & inference

d. Key technologies/frameworks
– Python, Jupyter Notebooks
– OpenCV, pytesseract
– pandas, regex for text processing
– spaCy for NER training & inference
– File‐based config (cfg)

## Diagram (Mermaid)

```mermaid
graph TD
    %% Input Layer
    subgraph "1. Input Layer"
        DI["Document Ingestion"]:::io
    end

    %% OCR Layer
    subgraph "2. OCR Layer"
        OCR1["OCR Notebook"]:::processing
        OCR2["Tesseract Inference"]:::processing
    end

    %% Data Prep Layer
    subgraph "3. Data Prep Layer"
        DP1["Data Prep Notebook"]:::processing
        DP2["Pickle Datasets"]:::storage
    end

    %% Preprocessing Layer
    subgraph "4. Preprocessing Layer"
        PREP1["Preprocessing Script"]:::processing
        PREP2["Preprocessing Notebook"]:::processing
    end

    %% Training Layer
    subgraph "5. Training Layer"
        CFG["Config Files"]:::storage
        SPACY_DATA["Training Data (.spacy)"]:::storage
        TRAIN_NB["Training Notebook"]:::model
        MODEL_BEST["Model Artifacts (best)"]:::storage
        MODEL_LAST["Model Artifacts (last)"]:::storage
    end

    %% Inference Layer
    subgraph "6. Inference Layer"
        INF_SCRIPT["Inference Script"]:::model
        INF_NB["Inference Notebook"]:::model
    end

    %% Storage Layer
    subgraph "7. Storage Layer"
        RAW["Raw & Preprocessed Data"]:::storage
        MODELS["Model Artifacts Storage"]:::storage
    end

    %% Connections
    DI --> OCR1
    DI --> OCR2
    OCR1 --> DP1
    DP1 --> DP2
    DP1 --> PREP1
    PREP1 --> PREP2
    PREP2 --> TRAIN_NB
    PREP2 --> INF_SCRIPT
    CFG --> TRAIN_NB
    SPACY_DATA --> TRAIN_NB
    TRAIN_NB --> MODEL_BEST
    TRAIN_NB --> MODEL_LAST
    MODEL_BEST --> INF_SCRIPT
    INF_SCRIPT --> INF_NB
    DP2 --> RAW
    MODEL_BEST --> MODELS
    MODEL_LAST --> MODELS

    %% Click Events
    click DI "https://github.com/patrykbala/documentner/blob/main/01_Pytesseract.ipynb"
    click OCR1 "https://github.com/patrykbala/documentner/blob/main/.ipynb_checkpoints/01_Pytesseract-checkpoint.ipynb"
    click OCR2 "https://github.com/patrykbala/documentner/blob/main/predictions.py"
    click DP1 "https://github.com/patrykbala/documentner/blob/main/02_Data_Preparation.ipynb"
    click DP2 "https://github.com/patrykbala/documentner/blob/main/data/TrainData.pickle"
    click PREP1 "https://github.com/patrykbala/documentner/blob/main/preprocess.py"
    click PREP2 "https://github.com/patrykbala/documentner/blob/main/03_Data_Preprocessing.ipynb"
    click CFG "https://github.com/patrykbala/documentner/blob/main/config.cfg"
    click SPACY_DATA "https://github.com/patrykbala/documentner/blob/main/data/train.spacy"
    click TRAIN_NB "https://github.com/patrykbala/documentner/blob/main/04_Predictions.ipynb"
    click MODEL_BEST "https://github.com/patrykbala/documentner/tree/main/output/model-best/"
    click MODEL_LAST "https://github.com/patrykbala/documentner/tree/main/output/model-last/"
    click INF_SCRIPT "https://github.com/patrykbala/documentner/blob/main/predictions.py"
    click INF_NB "https://github.com/patrykbala/documentner/blob/main/05_Final_predictions.ipynb"
    click RAW "https://github.com/patrykbala/documentner/tree/main/data/"
    click MODELS "https://github.com/patrykbala/documentner/tree/main/output/model-best/"

    %% Styles
    classDef io fill:#ADD8E6,stroke:#000;
    classDef processing fill:#90EE90,stroke:#000;
    classDef model fill:#FFA500,stroke:#000;
    classDef storage fill:#D3D3D3,stroke:#000;
```
