How sivkri/ner-product-extractor works
Project type and purpose
• It combines rule-based (regex + handcrafted NLP rules) and ML classification (TF-IDF + Logistic Regression) within a Dockerized Python environment.
File structure
• Top-level scripts:
• main.py – Orchestrates the full pipeline (ingestion → extraction → classification → output).
• information_extraction.ipynb – Exploratory notebook for prototyping/explaining rules.
• test_main.py, test_product_extraction.py – pytest suites validating end-to-end logic and individual extraction functions.
• Dockerfile.txt – Containerizes the app.
• requirements.txt – Lists Python dependencies (scikit-learn, pandas, regex/NLP libs).
• Data artifacts:
• ds_ner_test_case.csv (raw input)
• *_cleaned_combined.csv, enriched_products.csv (intermediate/ output)
• Additional:
• TASK Data Science – NER use case.pdf – Domain spec.
Key system components
a. Data Ingestion
• Reads raw CSV (ds_ner_test_case.csv) via pandas.
b. Preprocessing
• Cleans text, normalizes whitespace, lowercasing.
c. Rule-based NER Module
d. ML Classification Module
e. Standardization & Enrichment
• Maps raw extractions + predicted classes to standardized labels; outputs enriched_products.csv.
f. Orchestrator
• main.py ties all modules, handles I/O, writes final CSV.
g. Environment & Deployment
• Docker container builds via Dockerfile.txt for reproducibility.
h. Testing Layer
• Pytest scripts validate both rule-based and ML outputs.
Relationships & data flow
• Raw CSV → Ingestion → Preprocessing → Split path:
• Path 1: Rule-based NER → Standardizer
• Path 2: TF-IDF + ML classifiers → Standardizer
→ Merge & Enrich → Final CSV.
• Tests hook into modules at rule-based and end-to-end levels.
• Docker wraps the entire pipeline for reproducibility.
Architectural patterns & principles
• Pipeline pattern: clear sequential stages (ingest → extract → enrich).
• Separation of concerns: rule vs. ML extraction separated into modules.
• Configuration as code: Dockerfile + requirements.txt ensure environment consistency.
• Test-driven components: pytest enforces modular correctness.