# politicaldog/prototipo-pricescraper-ss architecture

> Architecture diagram of the GitHub repository politicaldog/prototipo-pricescraper-ss, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/politicaldog/prototipo-pricescraper-ss
- Repository: https://github.com/politicaldog/prototipo-pricescraper-ss

## Overview

PriceScraper ingests supermarket brochures, extracts and classifies product prices, loads records into PostgreSQL, and presents them for analysis and source verification. The documented main workflow is scraper → vision → NLP → load → dashboard; each stage exchanges files on disk, so stages can be rerun independently. A separate evaluation subsystem compares pipeline output with manually labeled ground truth. The README establishes the stage order and capabilities; sampled source excerpts and the source index support only the more specific internal calls shown below.

Main entry points: [`dashboard/queries.py`](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/dashboard/queries.py) (Dashboard queries), [`load/load.py`](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/load/load.py) (Record construction and load), [`evaluador/__main__.py`](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/evaluador/__main__.py) (Evaluation runner), [`nlp/regex_extractor.py`](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/nlp/regex_extractor.py) (Block classification)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_acquisition["Brochure acquisition"]
  node_scraper_sources["Source scrapers<br/>[tiendeo.py]"]
  node_downloader["Page downloader<br/>[downloader.py]"]
end

subgraph group_extraction["Image and text processing"]
  node_image_prep["Image preprocessing<br/>[preprocessor.py]"]
  node_ocr["OCR engine<br/>[ocr_engine.py]"]
  node_regions["Product regions<br/>[region_detector.py]"]
  node_classification["Block classification<br/>[regex_extractor.py]"]
  node_catalog["Product catalog"]
  node_normalization["Name normalization<br/>[normalizador.py]"]
end

subgraph group_persistence["Price data"]
  node_load["Record construction and load<br/>[load.py]"]
  node_postgres[("PostgreSQL")]
end

subgraph group_dashboard["Dashboard"]
  node_dashboard_views["Web views<br/>[views.py]"]
  node_dashboard_queries["Dashboard queries<br/>[queries.py]"]
  node_origin["Source viewer<br/>[origen.py]"]
  node_promotions["Promotion views<br/>[promociones.py]"]
  node_reliability["Reliability metrics<br/>[confiabilidad.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_eval_entry["Evaluation runner<br/>[__main__.py]"]
  node_ground_truth["Labeled ground truth<br/>[gt.py]"]
  node_eval_pipeline["Pipeline reconstruction<br/>[pipeline.py]"]
  node_comparison["Record comparison<br/>[comparar.py]"]
  node_eval_metrics["Evaluation metrics<br/>[metricas.py]"]
end

node_retailer_sources(("Retailer sites"))
node_analyst(("Dashboard user"))

node_retailer_sources -->|"provide brochures"| node_scraper_sources
node_scraper_sources -.->|"download pages"| node_downloader
node_downloader -->|"page images"| node_image_prep
node_image_prep -->|"prepare images"| node_ocr
node_ocr -->|"OCR blocks"| node_classification
node_classification -.->|"classify products"| node_catalog
node_classification -.->|"classified names"| node_normalization
node_classification -->|"NLP results"| node_load
node_normalization -->|"normalize names"| node_load
node_regions -->|"ROI candidates"| node_load
node_load -->|"write records"| node_postgres
node_postgres -->|"query records"| node_dashboard_queries
node_analyst -.->|"request pages"| node_dashboard_views
node_dashboard_views -->|"request data"| node_dashboard_queries
node_dashboard_views -->|"request source detail"| node_origin
node_dashboard_views -->|"request promotions"| node_promotions
node_dashboard_queries -->|"use source details"| node_origin
node_dashboard_queries -->|"combine promotions"| node_promotions
node_dashboard_queries -->|"compute reliability"| node_reliability
node_eval_entry -->|"load labels"| node_ground_truth
node_eval_entry -->|"build pipeline records"| node_eval_pipeline
node_eval_pipeline -->|"reuse record builder"| node_load
node_eval_pipeline -->|"normalize records"| node_normalization
node_eval_pipeline -->|"get ROI candidates"| node_regions
node_eval_entry -->|"compare results"| node_comparison
node_comparison -->|"compare labels"| node_ground_truth
node_eval_entry -->|"calculate metrics"| node_eval_metrics
node_eval_metrics -->|"aggregate comparisons"| node_comparison

click node_scraper_sources "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/scraper/sources/tiendeo.py"
click node_downloader "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/scraper/downloader.py"
click node_image_prep "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/vision/preprocessor.py"
click node_ocr "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/vision/ocr_engine.py"
click node_regions "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/vision/region_detector.py"
click node_classification "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/nlp/regex_extractor.py"
click node_catalog "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/nlp/catalogo_productos.py"
click node_normalization "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/nlp/normalizador.py"
click node_load "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/load/load.py"
click node_dashboard_views "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/dashboard/views.py"
click node_dashboard_queries "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/dashboard/queries.py"
click node_origin "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/dashboard/origen.py"
click node_promotions "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/dashboard/promociones.py"
click node_reliability "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/dashboard/confiabilidad.py"
click node_eval_entry "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/evaluador/__main__.py"
click node_ground_truth "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/evaluador/gt.py"
click node_eval_pipeline "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/evaluador/pipeline.py"
click node_comparison "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/evaluador/comparar.py"
click node_eval_metrics "https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/evaluador/metricas.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scraper_sources,node_downloader,node_analyst toneBlue
class node_image_prep,node_ocr,node_regions,node_classification,node_catalog,node_normalization toneAmber
class node_load,node_postgres toneMint
class node_dashboard_views,node_dashboard_queries,node_origin,node_promotions,node_reliability toneRose
class node_eval_entry,node_ground_truth,node_eval_pipeline,node_comparison,node_eval_metrics,node_retailer_sources toneIndigo
```

## Components

### Brochure acquisition

- [Source scrapers](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/scraper/sources/tiendeo.py): `scraper/sources/tiendeo.py`
- [Page downloader](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/scraper/downloader.py): `scraper/downloader.py`

### Image and text processing

- [Image preprocessing](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/vision/preprocessor.py): `vision/preprocessor.py`
- [OCR engine](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/vision/ocr_engine.py): `vision/ocr_engine.py`
- [Product regions](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/vision/region_detector.py): `vision/region_detector.py`
- [Block classification](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/nlp/regex_extractor.py): `nlp/regex_extractor.py`
- [Product catalog](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/nlp/catalogo_productos.py): `nlp/catalogo_productos.py`
- [Name normalization](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/nlp/normalizador.py): `nlp/normalizador.py`

### Price data

- [Record construction and load](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/load/load.py): `load/load.py`
- **PostgreSQL**

### Dashboard

- [Web views](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/dashboard/views.py): `dashboard/views.py`
- [Dashboard queries](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/dashboard/queries.py): `dashboard/queries.py`
- [Source viewer](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/dashboard/origen.py): `dashboard/origen.py`
- [Promotion views](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/dashboard/promociones.py): `dashboard/promociones.py`
- [Reliability metrics](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/dashboard/confiabilidad.py): `dashboard/confiabilidad.py`

### Evaluation

- [Evaluation runner](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/evaluador/__main__.py): `evaluador/__main__.py`
- [Labeled ground truth](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/evaluador/gt.py): `evaluador/gt.py`
- [Pipeline reconstruction](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/evaluador/pipeline.py): `evaluador/pipeline.py`
- [Record comparison](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/evaluador/comparar.py): `evaluador/comparar.py`
- [Evaluation metrics](https://github.com/politicaldog/prototipo-pricescraper-ss/blob/main/evaluador/metricas.py): `evaluador/metricas.py`

### Other components

- **Retailer sites**
- **Dashboard user**

## Connections

- Retailer sites → Source scrapers: provide brochures (evidence: `README.md`)
- Source scrapers → Page downloader: download pages
- Page downloader → Image preprocessing: page images (evidence: `README.md`)
- Image preprocessing → OCR engine: prepare images (evidence: `README.md`)
- OCR engine → Block classification: OCR blocks (evidence: `README.md`)
- Block classification → Product catalog: classify products (evidence: `nlp/regex_extractor.py`)
- Block classification → Name normalization: classified names
- Block classification → Record construction and load: NLP results (evidence: `README.md`)
- Name normalization → Record construction and load: normalize names (evidence: `evaluador/pipeline.py`)
- Product regions → Record construction and load: ROI candidates (evidence: `load/load.py`)
- Record construction and load → PostgreSQL: write records (evidence: `README.md`)
- PostgreSQL → Dashboard queries: query records (evidence: `README.md`)
- Dashboard user → Web views: request pages
- Web views → Dashboard queries: request data (evidence: `dashboard/views.py`)
- Web views → Source viewer: request source detail (evidence: `dashboard/views.py`)
- Web views → Promotion views: request promotions (evidence: `dashboard/views.py`)
- Dashboard queries → Source viewer: use source details (evidence: `dashboard/queries.py`)
- Dashboard queries → Promotion views: combine promotions (evidence: `dashboard/queries.py`)
- Dashboard queries → Reliability metrics: compute reliability (evidence: `dashboard/queries.py`)
- Evaluation runner → Labeled ground truth: load labels (evidence: `evaluador/__main__.py`)
- Evaluation runner → Pipeline reconstruction: build pipeline records (evidence: `evaluador/__main__.py`)
- Pipeline reconstruction → Record construction and load: reuse record builder (evidence: `evaluador/pipeline.py`)
- Pipeline reconstruction → Name normalization: normalize records (evidence: `evaluador/pipeline.py`)
- Pipeline reconstruction → Product regions: get ROI candidates (evidence: `evaluador/pipeline.py`)
- Evaluation runner → Record comparison: compare results (evidence: `evaluador/__main__.py`)
- Record comparison → Labeled ground truth: compare labels (evidence: `evaluador/comparar.py`)
- Evaluation runner → Evaluation metrics: calculate metrics (evidence: `evaluador/__main__.py`)
- Evaluation metrics → Record comparison: aggregate comparisons (evidence: `evaluador/metricas.py`)
