How satyaa133/aitoolspipeline works
This is a Python pipeline for curating an AI-tools catalogue into a validated CSV/XLSX dataset. Its principal workflow starts when the runner loads or harvests candidates, then verifies websites, deduplicates, enriches, generates descriptions, scores, validates, and exports records. Normalization is a shared internal layer used by verification and deduplication. The README documents broad curation capabilities; the excerpts show the runner’s orchestration and core transformations, but not every implementation detail. Edges reflect those calls, data transfers, and documented behaviors.
Main entry points: run_pipeline.py Pipeline runner · verify.py Website verification · discovery.py Candidate discovery · dedupe.py Duplicate resolution
Components 15
Candidate Ingestion
- Pipeline runner
scripts/run_pipeline.py - Candidate discovery
src/discovery.py - Raw candidates
data/raw/discovery_candidates.csv
Record Curation
- Website verification
src/verify.py - URL and name normalization
src/normalize.py - Duplicate resolution
src/dedupe.py - Metadata enrichment
src/enrich.py - Description generation
src/llm_descriptions.py
Scoring and QA
- Curation scoring
src/scoring.py - Final dataset schema
src/schema.py - Quality validation
src/qa.py
Dataset Outputs
- Processed candidates
data/processed - Curated dataset
data/final
Other components
- Pipeline operator
- Google favicon endpoint
Connections 17
- Pipeline operator → Pipeline runner: runs
README.md - Pipeline runner → Candidate discovery: calls if needed
scripts/run_pipeline.py - Candidate discovery → Raw candidates: writes candidates
src/discovery.py - Pipeline runner → Raw candidates: loads candidates
scripts/run_pipeline.py - Pipeline runner → Website verification: calls per record
scripts/run_pipeline.py - Website verification → URL and name normalization: canonicalizes URLs
src/verify.py - Website verification → Google favicon endpoint: constructs logo URL
src/verify.py - Pipeline runner → Duplicate resolution: calls
scripts/run_pipeline.py - Duplicate resolution → URL and name normalization: normalizes names and domains
src/dedupe.py - Pipeline runner → Metadata enrichment: calls per record
scripts/run_pipeline.py - Pipeline runner → Description generation: calls per record
scripts/run_pipeline.py - Pipeline runner → Curation scoring: scores records
scripts/run_pipeline.py - Curation scoring → Processed candidates: runner writes scored rows
- Pipeline runner → Final dataset schema: aligns columns
scripts/run_pipeline.py - Pipeline runner → Quality validation: validates and reports
scripts/run_pipeline.py - Quality validation → Final dataset schema: checks final columns
src/qa.py - Pipeline runner → Curated dataset: exports CSV and Excel
scripts/run_pipeline.py