# sivkri/ner-product-extractor architecture

> Architecture diagram of the GitHub repository sivkri/ner-product-extractor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sivkri/ner-product-extractor
- Repository: https://github.com/sivkri/ner-product-extractor

## Overview

##### Project type and purpose
- It combines rule-based (regex + handcrafted NLP rules) and ML classification (TF-IDF + Logistic Regression) within a Dockerized Python environment.

##### File structure
- Top-level scripts:
- main.py           – Orchestrates the full pipeline (ingestion → extraction → classification → output).
- information_extraction.ipynb – Exploratory notebook for prototyping/explaining rules.
- test_main.py, test_product_extraction.py – pytest suites validating end-to-end logic and individual extraction functions.
- Dockerfile.txt    – Containerizes the app.
- requirements.txt  – Lists Python dependencies (scikit-learn, pandas, regex/NLP libs).
- Data artifacts:
- ds_ner_test_case.csv (raw input)
- *_cleaned_combined.csv, enriched_products.csv (intermediate/ output)
- Additional:
- TASK Data Science – NER use case.pdf – Domain spec.

##### Key system components
a. Data Ingestion
- Reads raw CSV (`ds_ner_test_case.csv`) via pandas.
b. Preprocessing
- Cleans text, normalizes whitespace, lowercasing.
c. Rule-based NER Module
d. ML Classification Module
e. Standardization & Enrichment
- Maps raw extractions + predicted classes to standardized labels; outputs enriched_products.csv.
f. Orchestrator
- `main.py` ties all modules, handles I/O, writes final CSV.
g. Environment & Deployment
- Docker container builds via Dockerfile.txt for reproducibility.
h. Testing Layer
- Pytest scripts validate both rule-based and ML outputs.

##### Relationships & data flow
- Raw CSV → Ingestion → Preprocessing → Split path:
- Path 1: Rule-based NER → Standardizer
- Path 2: TF-IDF + ML classifiers → Standardizer
→ Merge & Enrich → Final CSV.
- Tests hook into modules at rule-based and end-to-end levels.
- Docker wraps the entire pipeline for reproducibility.

##### Architectural patterns & principles
- Pipeline pattern: clear sequential stages (ingest → extract → enrich).
- Separation of concerns: rule vs. ML extraction separated into modules.
- Configuration as code: Dockerfile + requirements.txt ensure environment consistency.
- Test-driven components: pytest enforces modular correctness.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Store
    RawCSV[("ds_ner_test_case.csv")]:::dataStore
    click RawCSV "https://github.com/sivkri/ner-product-extractor/blob/main/ds_ner_test_case.csv"

    %% Docker Container Boundary
    subgraph "Docker Container\n(Dockerfile.txt / requirements.txt)" 
        style DockerContainer stroke-dasharray: 5 5,stroke:#888,fill:transparent
        %% Orchestrator
        Orchestrator["Orchestrator\n(main.py)"]:::core
        click Orchestrator "https://github.com/sivkri/ner-product-extractor/blob/main/main.py"

        %% Preprocessing
        Preprocessing["Preprocessing"]:::core

        %% Parallel Extraction
        RuleNER["Rule-based NER"]:::core
        TFIDF["TF-IDF + Logistic Regression"]:::ml

        %% Standardization & Enrichment
        Standardizer["Standardization &\nEnrichment"]:::core
        click Standardizer "https://github.com/sivkri/ner-product-extractor/blob/main/main.py"

        %% Output
        EnrichedCSV[("enriched_products.csv")]:::dataStore
        click EnrichedCSV "https://github.com/sivkri/ner-product-extractor/blob/main/enriched_products.csv"
    end

    %% Flow
    RawCSV -->|"CSV Input"| Orchestrator
    Orchestrator -->|"DataFrame"| Preprocessing
    Preprocessing -->|"Cleaned DataFrame"| RuleNER
    Preprocessing -->|"Cleaned DataFrame"| TFIDF
    TFIDF -->|"TF-IDF Vectors"| Standardizer
    RuleNER -->|"Extracted Entities"| Standardizer
    Standardizer -->|"Enriched CSV"| EnrichedCSV

    %% Notebook and Tests
    Notebook["information_extraction.ipynb"]:::notebook
    click Notebook "https://github.com/sivkri/ner-product-extractor/blob/main/information_extraction.ipynb"
    Notebook -.->|dotted| Preprocessing
    Notebook -.->|dotted| RuleNER

    TestE2E["test_main.py"]:::test
    click TestE2E "https://github.com/sivkri/ner-product-extractor/blob/main/test_main.py"
    TestE2E -.->|dotted| Orchestrator

    TestUnit["test_product_extraction.py"]:::test
    click TestUnit "https://github.com/sivkri/ner-product-extractor/blob/main/test_product_extraction.py"
    TestUnit -.->|dotted| RuleNER
    TestUnit -.->|dotted| TFIDF

    %% Styles
    classDef dataStore fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef core fill:#90EE90,stroke:#000,stroke-width:1px
    classDef ml fill:#FFA500,stroke:#000,stroke-width:1px
    classDef notebook fill:#E0E0E0,stroke:#000,stroke-width:1px,stroke-dasharray: 2 2
    classDef test fill:#E0E0E0,stroke:#000,stroke-width:1px,stroke-dasharray: 2 2
```
