How politicaldog/prototipo-pricescraper-ss works
PriceScraper ingests supermarket brochures, extracts and classifies product prices, loads records into PostgreSQL, and presents them for analysis and source verification. The documented main workflow is scraper → vision → NLP → load → dashboard; each stage exchanges files on disk, so stages can be rerun independently. A separate evaluation subsystem compares pipeline output with manually labeled ground truth. The README establishes the stage order and capabilities; sampled source excerpts and the source index support only the more specific internal calls shown below.
Main entry points: queries.py Dashboard queries · load.py Record construction and load · __main__.py Evaluation runner · regex_extractor.py Block classification
Components 22
Brochure acquisition
- Source scrapers
scraper/sources/tiendeo.py - Page downloader
scraper/downloader.py
Image and text processing
- Image preprocessing
vision/preprocessor.py - OCR engine
vision/ocr_engine.py - Product regions
vision/region_detector.py - Block classification
nlp/regex_extractor.py - Product catalog
nlp/catalogo_productos.py - Name normalization
nlp/normalizador.py
Price data
- Record construction and load
load/load.py - PostgreSQL
Dashboard
- Web views
dashboard/views.py - Dashboard queries
dashboard/queries.py - Source viewer
dashboard/origen.py - Promotion views
dashboard/promociones.py - Reliability metrics
dashboard/confiabilidad.py
Evaluation
- Evaluation runner
evaluador/__main__.py - Labeled ground truth
evaluador/gt.py - Pipeline reconstruction
evaluador/pipeline.py - Record comparison
evaluador/comparar.py - Evaluation metrics
evaluador/metricas.py
Other components
- Retailer sites
- Dashboard user
Connections 28
- Retailer sites → Source scrapers: provide brochures
README.md - Source scrapers → Page downloader: download pages
- Page downloader → Image preprocessing: page images
README.md - Image preprocessing → OCR engine: prepare images
README.md - OCR engine → Block classification: OCR blocks
README.md - Block classification → Product catalog: classify products
nlp/regex_extractor.py - Block classification → Name normalization: classified names
- Block classification → Record construction and load: NLP results
README.md - Name normalization → Record construction and load: normalize names
evaluador/pipeline.py - Product regions → Record construction and load: ROI candidates
load/load.py - Record construction and load → PostgreSQL: write records
README.md - PostgreSQL → Dashboard queries: query records
README.md - Dashboard user → Web views: request pages
- Web views → Dashboard queries: request data
dashboard/views.py - Web views → Source viewer: request source detail
dashboard/views.py - Web views → Promotion views: request promotions
dashboard/views.py - Dashboard queries → Source viewer: use source details
dashboard/queries.py - Dashboard queries → Promotion views: combine promotions
dashboard/queries.py - Dashboard queries → Reliability metrics: compute reliability
dashboard/queries.py - Evaluation runner → Labeled ground truth: load labels
evaluador/__main__.py - Evaluation runner → Pipeline reconstruction: build pipeline records
evaluador/__main__.py - Pipeline reconstruction → Record construction and load: reuse record builder
evaluador/pipeline.py - Pipeline reconstruction → Name normalization: normalize records
evaluador/pipeline.py - Pipeline reconstruction → Product regions: get ROI candidates
evaluador/pipeline.py - Evaluation runner → Record comparison: compare results
evaluador/__main__.py - Record comparison → Labeled ground truth: compare labels
evaluador/comparar.py - Evaluation runner → Evaluation metrics: calculate metrics
evaluador/__main__.py - Evaluation metrics → Record comparison: aggregate comparisons
evaluador/metricas.py