How mohamedshakir3/find-my-professor-scraper works
This repository collects university faculty directories and professor profiles, extracts research details, and supports embedding and search-oriented data workflows. The principal path is directory discovery, profile fetching and extraction, then persistence or JSONL export; GPU results can be imported and embedded. University-specific scrapers, cloud processing, and orchestration are separate capabilities. The README is empty, and several modules are unsampled, so links are limited to relationships shown in excerpts or the source index.
Main entry points: scrape_new_unis.py Scrape and export · export_for_gpu.py GPU export · connection.py Connection pool · import_from_gpu.py GPU result import
Components 18
Directory collection
- Workflow orchestrator
orchestrator.py - Scrape and export
scrape_new_unis.py - Generic directory scraper
universities/generic.py - University-specific scrapers
universities - Scraper interface
core/interfaces.py
Profile processing
- Profile processor
pipeline/profile_processor.py - Cloud worker
pipeline/cloud_worker.py - Scholar workflow
scholar.py
Data and embeddings
- PostgreSQL database
- Connection pool
db/connection.py - Professor repositories
db/repositories.py - Embedding pipeline
pipeline/embedder.py
Batch exchange
- GPU export
export_for_gpu.py - GPU result import
import_from_gpu.py - JSON database export
db/export_json.py - JSON database import
db/import_json.py
Other components
- Operator
- GPU processing service
Connections 19
- Operator → Workflow orchestrator: runs
- Operator → Scrape and export: runs
- Scrape and export → Generic directory scraper: discovers profiles
scrape_new_unis.py - Generic directory scraper → Scraper interface: extends
universities/generic.py - University-specific scrapers → Scraper interface: implements
universities/uoft.py - Scrape and export → Profile processor: fetches profiles
scrape_new_unis.py - Scrape and export → GPU export: exports records
- Connection pool → PostgreSQL database: connects
db/connection.py - Professor repositories → Connection pool: gets connections
db/repositories.py - Professor repositories → PostgreSQL database: queries
db/repositories.py - GPU export → Connection pool: reads records
export_for_gpu.py - GPU export → Profile processor: fetches profiles
export_for_gpu.py - GPU export → GPU processing service: sends JSONL
- GPU processing service → GPU result import: returns results
- GPU result import → Embedding pipeline: embeds results
import_from_gpu.py - GPU result import → Connection pool: updates records
import_from_gpu.py - Embedding pipeline → PostgreSQL database: reads and writes
pipeline/embedder.py - JSON database export → PostgreSQL database: exports data
- JSON database import → PostgreSQL database: imports data