# princeg0210/autonomous-catalog-enrichment architecture

> Architecture diagram of the GitHub repository princeg0210/autonomous-catalog-enrichment, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/princeg0210/autonomous-catalog-enrichment
- Repository: https://github.com/princeg0210/autonomous-catalog-enrichment

## Overview

This repository implements a product-data enrichment pipeline with a small dashboard/API front end. The principal workflow accepts a SKU, dispatches asynchronous enrichment, builds an isolated product context, discovers and classifies source data, then validates and persists enriched results or review exceptions. The README documents additional source tiers and syndication capabilities beyond the sampled implementation. Edges use excerpts, SOURCE INDEX references, or README statements; unsampled subsystem details and integrations are not expanded beyond that evidence.

Main entry points: [`tasks.py`](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/tasks.py) (Async Enrichment), [`discovery.py`](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/discovery.py) (Source Discovery), [`main.py`](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/main.py) (API Gateway), [`models.py`](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/models.py) (Database Operations)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingress["Ingestion &amp; UI"]
  node_gateway["API Gateway<br/>[main.py]"]
  node_dashboard["Web Dashboard<br/>[index.html]"]
  node_cache[("Redis Cache")]
end

subgraph group_enrichment["Enrichment Pipeline"]
  node_worker["Async Enrichment<br/>[tasks.py]"]
  node_context["Product Context<br/>[context.py]"]
  node_discovery["Source Discovery<br/>[discovery.py]"]
  node_source_data["Catalog Dataset<br/>[dataset.csv]"]
  node_web_sources["Product Sources"]
  node_atomization["Attribute Classification"]
  node_normalizer["Unit Normalization<br/>[normalizer.py]"]
  node_dedup["Attribute Deduplication<br/>[deduplicator.py]"]
  node_taxonomy["Taxonomy Classification<br/>[taxonomy.py]"]
end

subgraph group_quality["Quality &amp; Syndication"]
  node_copywriter["Channel Copywriting<br/>[copywriter.py]"]
  node_quality_gate["Quality Gate<br/>[quality_gate.py]"]
end

subgraph group_persistence["Persistence &amp; Review"]
  node_models["Database Operations<br/>[models.py]"]
  node_postgres[("Product Database")]
  node_hitl["HITL Queue"]
end

node_operator(("Catalog Operator"))

node_operator -->|"submits SKU"| node_gateway
node_gateway -->|"checks cache"| node_cache
node_gateway -->|"dispatches job"| node_worker
node_gateway -->|"serves dashboard"| node_dashboard
node_dashboard -.->|"requests data"| node_gateway
node_worker -->|"checks cache"| node_cache
node_worker -->|"creates context"| node_context
node_worker -->|"runs discovery"| node_discovery
node_discovery -->|"reads records"| node_source_data
node_discovery -->|"fetches product page"| node_web_sources
node_discovery -->|"classifies attributes"| node_atomization
node_discovery -->|"normalizes values"| node_normalizer
node_discovery -->|"deduplicates attributes"| node_dedup
node_worker -->|"classifies product"| node_taxonomy
node_worker -->|"generates descriptions"| node_copywriter
node_worker -->|"evaluates quality"| node_quality_gate
node_worker -->|"persists results"| node_models
node_models -->|"reads and writes"| node_postgres
node_worker -->|"records exceptions"| node_hitl
node_gateway -->|"lists review items"| node_models

click node_gateway "https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/main.py"
click node_dashboard "https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/static/index.html"
click node_worker "https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/tasks.py"
click node_context "https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/context.py"
click node_discovery "https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/discovery.py"
click node_source_data "https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/dataset.csv"
click node_atomization "https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/attribute_classifier.py"
click node_normalizer "https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/normalizer.py"
click node_dedup "https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/deduplicator.py"
click node_taxonomy "https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/taxonomy.py"
click node_copywriter "https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/copywriter.py"
click node_quality_gate "https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/quality_gate.py"
click node_models "https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/models.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_gateway,node_dashboard,node_cache toneBlue
class node_worker,node_context,node_discovery,node_source_data,node_web_sources,node_atomization,node_normalizer,node_dedup,node_taxonomy toneAmber
class node_copywriter,node_quality_gate toneMint
class node_models,node_postgres,node_hitl toneRose
class node_operator toneIndigo
```

## Components

### Ingestion & UI

- [API Gateway](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/main.py): `main.py`
- [Web Dashboard](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/static/index.html): `static/index.html`
- **Redis Cache**

### Enrichment Pipeline

- [Async Enrichment](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/tasks.py): `tasks.py`
- [Product Context](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/context.py): `context.py`
- [Source Discovery](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/discovery.py): `discovery.py`
- [Catalog Dataset](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/dataset.csv): `dataset.csv`
- **Product Sources**
- [Attribute Classification](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/attribute_classifier.py): `attribute_classifier.py`
- [Unit Normalization](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/normalizer.py): `normalizer.py`
- [Attribute Deduplication](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/deduplicator.py): `deduplicator.py`
- [Taxonomy Classification](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/taxonomy.py): `taxonomy.py`

### Quality & Syndication

- [Channel Copywriting](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/copywriter.py): `copywriter.py`
- [Quality Gate](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/quality_gate.py): `quality_gate.py`

### Persistence & Review

- [Database Operations](https://github.com/princeg0210/autonomous-catalog-enrichment/blob/main/models.py): `models.py`
- **Product Database**
- **HITL Queue**

### Other components

- **Catalog Operator**

## Connections

- Catalog Operator → API Gateway: submits SKU (evidence: `README.md`)
- API Gateway → Redis Cache: checks cache (evidence: `main.py`)
- API Gateway → Async Enrichment: dispatches job (evidence: `main.py`)
- API Gateway → Web Dashboard: serves dashboard (evidence: `main.py`)
- Web Dashboard → API Gateway: requests data
- Async Enrichment → Redis Cache: checks cache (evidence: `tasks.py`)
- Async Enrichment → Product Context: creates context (evidence: `tasks.py`)
- Async Enrichment → Source Discovery: runs discovery (evidence: `tasks.py`)
- Source Discovery → Catalog Dataset: reads records (evidence: `discovery.py`)
- Source Discovery → Product Sources: fetches product page (evidence: `discovery.py`)
- Source Discovery → Attribute Classification: classifies attributes (evidence: `discovery.py`)
- Source Discovery → Unit Normalization: normalizes values (evidence: `discovery.py`)
- Source Discovery → Attribute Deduplication: deduplicates attributes (evidence: `discovery.py`)
- Async Enrichment → Taxonomy Classification: classifies product (evidence: `tasks.py`)
- Async Enrichment → Channel Copywriting: generates descriptions (evidence: `tasks.py`)
- Async Enrichment → Quality Gate: evaluates quality (evidence: `tasks.py`)
- Async Enrichment → Database Operations: persists results (evidence: `tasks.py`)
- Database Operations → Product Database: reads and writes (evidence: `models.py`)
- Async Enrichment → HITL Queue: records exceptions (evidence: `tasks.py`)
- API Gateway → Database Operations: lists review items (evidence: `main.py`)
