# mohamedshakir3/find-my-professor-scraper architecture

> Architecture diagram of the GitHub repository mohamedshakir3/find-my-professor-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mohamedshakir3/find-my-professor-scraper
- Repository: https://github.com/mohamedshakir3/find-my-professor-scraper

## Overview

This repository collects university faculty directories and professor profiles, extracts research details, and supports embedding and search-oriented data workflows. The principal path is directory discovery, profile fetching and extraction, then persistence or JSONL export; GPU results can be imported and embedded. University-specific scrapers, cloud processing, and orchestration are separate capabilities. The README is empty, and several modules are unsampled, so links are limited to relationships shown in excerpts or the source index.

Main entry points: [`scrape_new_unis.py`](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/scrape_new_unis.py) (Scrape and export), [`export_for_gpu.py`](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/export_for_gpu.py) (GPU export), [`db/connection.py`](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/db/connection.py) (Connection pool), [`import_from_gpu.py`](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/import_from_gpu.py) (GPU result import)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Directory collection"]
  node_orchestrator["Workflow orchestrator<br/>[orchestrator.py]"]
  node_scrape_runner["Scrape and export<br/>[scrape_new_unis.py]"]
  node_generic_scraper["Generic directory scraper<br/>[generic.py]"]
  node_university_scrapers["University-specific scrapers"]
  node_scraper_interface["Scraper interface<br/>[interfaces.py]"]
end

subgraph group_processing["Profile processing"]
  node_profile_processor["Profile processor"]
  node_cloud_worker["Cloud worker<br/>[cloud_worker.py]"]
  node_scholar["Scholar workflow<br/>[scholar.py]"]
end

subgraph group_storage["Data and embeddings"]
  node_database[("PostgreSQL database")]
  node_connection_pool["Connection pool<br/>[connection.py]"]
  node_repositories["Professor repositories<br/>[repositories.py]"]
  node_embedder["Embedding pipeline<br/>[embedder.py]"]
end

subgraph group_exchange["Batch exchange"]
  node_gpu_export["GPU export<br/>[export_for_gpu.py]"]
  node_gpu_import["GPU result import<br/>[import_from_gpu.py]"]
  node_json_export["JSON database export<br/>[export_json.py]"]
  node_json_import["JSON database import<br/>[import_json.py]"]
end

node_operator(("Operator"))
node_gpu_service(("GPU processing service"))

node_operator -.->|"runs"| node_orchestrator
node_operator -.->|"runs"| node_scrape_runner
node_scrape_runner -->|"discovers profiles"| node_generic_scraper
node_generic_scraper -->|"extends"| node_scraper_interface
node_university_scrapers -->|"implements"| node_scraper_interface
node_scrape_runner -->|"fetches profiles"| node_profile_processor
node_scrape_runner -.->|"exports records"| node_gpu_export
node_connection_pool -->|"connects"| node_database
node_repositories -->|"gets connections"| node_connection_pool
node_repositories -->|"queries"| node_database
node_gpu_export -->|"reads records"| node_connection_pool
node_gpu_export -->|"fetches profiles"| node_profile_processor
node_gpu_export -.->|"sends JSONL"| node_gpu_service
node_gpu_service -.->|"returns results"| node_gpu_import
node_gpu_import -->|"embeds results"| node_embedder
node_gpu_import -->|"updates records"| node_connection_pool
node_embedder -->|"reads and writes"| node_database
node_json_export -.->|"exports data"| node_database
node_json_import -.->|"imports data"| node_database

click node_orchestrator "https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/orchestrator.py"
click node_scrape_runner "https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/scrape_new_unis.py"
click node_generic_scraper "https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/universities/generic.py"
click node_university_scrapers "https://github.com/mohamedshakir3/find-my-professor-scraper/tree/master/universities"
click node_scraper_interface "https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/core/interfaces.py"
click node_profile_processor "https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/pipeline/profile_processor.py"
click node_cloud_worker "https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/pipeline/cloud_worker.py"
click node_scholar "https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/scholar.py"
click node_connection_pool "https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/db/connection.py"
click node_repositories "https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/db/repositories.py"
click node_embedder "https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/pipeline/embedder.py"
click node_gpu_export "https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/export_for_gpu.py"
click node_gpu_import "https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/import_from_gpu.py"
click node_json_export "https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/db/export_json.py"
click node_json_import "https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/db/import_json.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_orchestrator,node_scrape_runner,node_generic_scraper,node_university_scrapers,node_scraper_interface toneBlue
class node_profile_processor,node_cloud_worker,node_scholar toneAmber
class node_database,node_connection_pool,node_repositories,node_embedder toneMint
class node_gpu_export,node_gpu_import,node_json_export,node_json_import toneRose
class node_operator,node_gpu_service toneIndigo
```

## Components

### Directory collection

- [Workflow orchestrator](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/orchestrator.py): `orchestrator.py`
- [Scrape and export](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/scrape_new_unis.py): `scrape_new_unis.py`
- [Generic directory scraper](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/universities/generic.py): `universities/generic.py`
- [University-specific scrapers](https://github.com/mohamedshakir3/find-my-professor-scraper/tree/master/universities): `universities`
- [Scraper interface](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/core/interfaces.py): `core/interfaces.py`

### Profile processing

- [Profile processor](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/pipeline/profile_processor.py): `pipeline/profile_processor.py`
- [Cloud worker](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/pipeline/cloud_worker.py): `pipeline/cloud_worker.py`
- [Scholar workflow](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/scholar.py): `scholar.py`

### Data and embeddings

- **PostgreSQL database**
- [Connection pool](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/db/connection.py): `db/connection.py`
- [Professor repositories](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/db/repositories.py): `db/repositories.py`
- [Embedding pipeline](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/pipeline/embedder.py): `pipeline/embedder.py`

### Batch exchange

- [GPU export](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/export_for_gpu.py): `export_for_gpu.py`
- [GPU result import](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/import_from_gpu.py): `import_from_gpu.py`
- [JSON database export](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/db/export_json.py): `db/export_json.py`
- [JSON database import](https://github.com/mohamedshakir3/find-my-professor-scraper/blob/master/db/import_json.py): `db/import_json.py`

### Other components

- **Operator**
- **GPU processing service**

## Connections

- Operator → Workflow orchestrator: runs
- Operator → Scrape and export: runs
- Scrape and export → Generic directory scraper: discovers profiles (evidence: `scrape_new_unis.py`)
- Generic directory scraper → Scraper interface: extends (evidence: `universities/generic.py`)
- University-specific scrapers → Scraper interface: implements (evidence: `universities/uoft.py`)
- Scrape and export → Profile processor: fetches profiles (evidence: `scrape_new_unis.py`)
- Scrape and export → GPU export: exports records
- Connection pool → PostgreSQL database: connects (evidence: `db/connection.py`)
- Professor repositories → Connection pool: gets connections (evidence: `db/repositories.py`)
- Professor repositories → PostgreSQL database: queries (evidence: `db/repositories.py`)
- GPU export → Connection pool: reads records (evidence: `export_for_gpu.py`)
- GPU export → Profile processor: fetches profiles (evidence: `export_for_gpu.py`)
- GPU export → GPU processing service: sends JSONL
- GPU processing service → GPU result import: returns results
- GPU result import → Embedding pipeline: embeds results (evidence: `import_from_gpu.py`)
- GPU result import → Connection pool: updates records (evidence: `import_from_gpu.py`)
- Embedding pipeline → PostgreSQL database: reads and writes (evidence: `pipeline/embedder.py`)
- JSON database export → PostgreSQL database: exports data
- JSON database import → PostgreSQL database: imports data
