# satyaa133/aitoolspipeline architecture

> Architecture diagram of the GitHub repository satyaa133/aitoolspipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/satyaa133/aitoolspipeline
- Repository: https://github.com/satyaa133/aitoolspipeline

## Overview

This is a Python pipeline for curating an AI-tools catalogue into a validated CSV/XLSX dataset. Its principal workflow starts when the runner loads or harvests candidates, then verifies websites, deduplicates, enriches, generates descriptions, scores, validates, and exports records. Normalization is a shared internal layer used by verification and deduplication. The README documents broad curation capabilities; the excerpts show the runner’s orchestration and core transformations, but not every implementation detail. Edges reflect those calls, data transfers, and documented behaviors.

Main entry points: [`scripts/run_pipeline.py`](https://github.com/satyaa133/aitoolspipeline/blob/main/scripts/run_pipeline.py) (Pipeline runner), [`src/verify.py`](https://github.com/satyaa133/aitoolspipeline/blob/main/src/verify.py) (Website verification), [`src/discovery.py`](https://github.com/satyaa133/aitoolspipeline/blob/main/src/discovery.py) (Candidate discovery), [`src/dedupe.py`](https://github.com/satyaa133/aitoolspipeline/blob/main/src/dedupe.py) (Duplicate resolution)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Candidate Ingestion"]
  node_runner["Pipeline runner<br/>[run_pipeline.py]"]
  node_discovery["Candidate discovery<br/>[discovery.py]"]
  node_raw["Raw candidates"]
end

subgraph group_curation["Record Curation"]
  node_verification["Website verification<br/>[verify.py]"]
  node_normalization["URL and name normalization<br/>[normalize.py]"]
  node_deduplication["Duplicate resolution<br/>[dedupe.py]"]
  node_enrichment["Metadata enrichment<br/>[enrich.py]"]
  node_descriptions["Description generation"]
end

subgraph group_assessment["Scoring and QA"]
  node_scoring["Curation scoring<br/>[scoring.py]"]
  node_schema["Final dataset schema<br/>[schema.py]"]
  node_qa["Quality validation<br/>[qa.py]"]
end

subgraph group_outputs["Dataset Outputs"]
  node_processed["Processed candidates"]
  node_final["Curated dataset"]
end

node_operator(("Pipeline operator"))
node_logo_service["Google favicon endpoint"]

node_operator -->|"runs"| node_runner
node_runner -->|"calls if needed"| node_discovery
node_discovery -->|"writes candidates"| node_raw
node_runner -->|"loads candidates"| node_raw
node_runner -->|"calls per record"| node_verification
node_verification -->|"canonicalizes URLs"| node_normalization
node_verification -->|"constructs logo URL"| node_logo_service
node_runner -->|"calls"| node_deduplication
node_deduplication -->|"normalizes names and domains"| node_normalization
node_runner -->|"calls per record"| node_enrichment
node_runner -->|"calls per record"| node_descriptions
node_runner -->|"scores records"| node_scoring
node_scoring -->|"runner writes scored rows"| node_processed
node_runner -->|"aligns columns"| node_schema
node_runner -->|"validates and reports"| node_qa
node_qa -->|"checks final columns"| node_schema
node_runner -->|"exports CSV and Excel"| node_final

click node_runner "https://github.com/satyaa133/aitoolspipeline/blob/main/scripts/run_pipeline.py"
click node_discovery "https://github.com/satyaa133/aitoolspipeline/blob/main/src/discovery.py"
click node_raw "https://github.com/satyaa133/aitoolspipeline/blob/main/data/raw/discovery_candidates.csv"
click node_verification "https://github.com/satyaa133/aitoolspipeline/blob/main/src/verify.py"
click node_normalization "https://github.com/satyaa133/aitoolspipeline/blob/main/src/normalize.py"
click node_deduplication "https://github.com/satyaa133/aitoolspipeline/blob/main/src/dedupe.py"
click node_enrichment "https://github.com/satyaa133/aitoolspipeline/blob/main/src/enrich.py"
click node_descriptions "https://github.com/satyaa133/aitoolspipeline/blob/main/src/llm_descriptions.py"
click node_scoring "https://github.com/satyaa133/aitoolspipeline/blob/main/src/scoring.py"
click node_schema "https://github.com/satyaa133/aitoolspipeline/blob/main/src/schema.py"
click node_qa "https://github.com/satyaa133/aitoolspipeline/blob/main/src/qa.py"
click node_processed "https://github.com/satyaa133/aitoolspipeline/tree/main/data/processed"
click node_final "https://github.com/satyaa133/aitoolspipeline/tree/main/data/final"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_discovery,node_raw toneBlue
class node_verification,node_normalization,node_deduplication,node_enrichment,node_descriptions toneAmber
class node_scoring,node_schema,node_qa toneMint
class node_processed,node_final toneRose
class node_operator,node_logo_service toneIndigo
```

## Components

### Candidate Ingestion

- [Pipeline runner](https://github.com/satyaa133/aitoolspipeline/blob/main/scripts/run_pipeline.py): `scripts/run_pipeline.py`
- [Candidate discovery](https://github.com/satyaa133/aitoolspipeline/blob/main/src/discovery.py): `src/discovery.py`
- [Raw candidates](https://github.com/satyaa133/aitoolspipeline/blob/main/data/raw/discovery_candidates.csv): `data/raw/discovery_candidates.csv`

### Record Curation

- [Website verification](https://github.com/satyaa133/aitoolspipeline/blob/main/src/verify.py): `src/verify.py`
- [URL and name normalization](https://github.com/satyaa133/aitoolspipeline/blob/main/src/normalize.py): `src/normalize.py`
- [Duplicate resolution](https://github.com/satyaa133/aitoolspipeline/blob/main/src/dedupe.py): `src/dedupe.py`
- [Metadata enrichment](https://github.com/satyaa133/aitoolspipeline/blob/main/src/enrich.py): `src/enrich.py`
- [Description generation](https://github.com/satyaa133/aitoolspipeline/blob/main/src/llm_descriptions.py): `src/llm_descriptions.py`

### Scoring and QA

- [Curation scoring](https://github.com/satyaa133/aitoolspipeline/blob/main/src/scoring.py): `src/scoring.py`
- [Final dataset schema](https://github.com/satyaa133/aitoolspipeline/blob/main/src/schema.py): `src/schema.py`
- [Quality validation](https://github.com/satyaa133/aitoolspipeline/blob/main/src/qa.py): `src/qa.py`

### Dataset Outputs

- [Processed candidates](https://github.com/satyaa133/aitoolspipeline/tree/main/data/processed): `data/processed`
- [Curated dataset](https://github.com/satyaa133/aitoolspipeline/tree/main/data/final): `data/final`

### Other components

- **Pipeline operator**
- **Google favicon endpoint**

## Connections

- Pipeline operator → Pipeline runner: runs (evidence: `README.md`)
- Pipeline runner → Candidate discovery: calls if needed (evidence: `scripts/run_pipeline.py`)
- Candidate discovery → Raw candidates: writes candidates (evidence: `src/discovery.py`)
- Pipeline runner → Raw candidates: loads candidates (evidence: `scripts/run_pipeline.py`)
- Pipeline runner → Website verification: calls per record (evidence: `scripts/run_pipeline.py`)
- Website verification → URL and name normalization: canonicalizes URLs (evidence: `src/verify.py`)
- Website verification → Google favicon endpoint: constructs logo URL (evidence: `src/verify.py`)
- Pipeline runner → Duplicate resolution: calls (evidence: `scripts/run_pipeline.py`)
- Duplicate resolution → URL and name normalization: normalizes names and domains (evidence: `src/dedupe.py`)
- Pipeline runner → Metadata enrichment: calls per record (evidence: `scripts/run_pipeline.py`)
- Pipeline runner → Description generation: calls per record (evidence: `scripts/run_pipeline.py`)
- Pipeline runner → Curation scoring: scores records (evidence: `scripts/run_pipeline.py`)
- Curation scoring → Processed candidates: runner writes scored rows
- Pipeline runner → Final dataset schema: aligns columns (evidence: `scripts/run_pipeline.py`)
- Pipeline runner → Quality validation: validates and reports (evidence: `scripts/run_pipeline.py`)
- Quality validation → Final dataset schema: checks final columns (evidence: `src/qa.py`)
- Pipeline runner → Curated dataset: exports CSV and Excel (evidence: `scripts/run_pipeline.py`)
