# ashutosh-kumar-rao/linkedin-crawler architecture

> Architecture diagram of the GitHub repository ashutosh-kumar-rao/linkedin-crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ashutosh-kumar-rao/linkedin-crawler
- Repository: https://github.com/ashutosh-kumar-rao/linkedin-crawler

## Overview

Lead Research Engine accepts a search query, creates a research job, and hands it to a worker for discovery, relevance ranking, deduplication, enrichment, confidence scoring, and persistence. The browser can follow progress over SSE, browse and filter saved results, inspect lead details, and export CSV or XLSX. The README documents demo and optional search/email providers; sampled source confirms the job, pipeline, progress, and export wiring, while several UI, provider, and storage internals are represented only at their documented module paths.

Main entry points: [`apps/worker/src/pipeline/runResearchJob.ts`](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/worker/src/pipeline/runResearchJob.ts) (Research orchestration), [`apps/web/lib/api.ts`](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/web/lib/api.ts) (API client), [`packages/database/src/index.ts`](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/database/src/index.ts) (Postgres data), [`apps/api/src/routes/research.ts`](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/api/src/routes/research.ts) (Research jobs and SSE)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_experience["Browser experience"]
  node_web["Search and results UI<br/>[page.tsx]"]
  node_webcomponents["Progress, filters, details<br/>[ProgressPanel.tsx]"]
  node_webresults["Results and exports UI<br/>[ResultsTable.tsx]"]
  node_webapi["API client<br/>[api.ts]"]
end

subgraph group_api["API and jobs"]
  node_server["Fastify server<br/>[server.ts]"]
  node_researchroutes["Research jobs and SSE<br/>[research.ts]"]
  node_leadroutes["Lead queries<br/>[leads.ts]"]
  node_exportroutes["Export endpoint<br/>[export.ts]"]
  node_queue["Job queue and events<br/>[index.ts]"]
end

subgraph group_research["Research pipeline"]
  node_worker["Queue worker<br/>[index.ts]"]
  node_pipeline["Research orchestration<br/>[runResearchJob.ts]"]
  node_verification["Ranking and dedup<br/>[index.ts]"]
  node_persist["Lead persistence<br/>[persist.ts]"]
  node_database[("Postgres data<br/>[index.ts]")]
end

subgraph group_integrations["Providers and exports"]
  node_discovery["Candidate discovery<br/>[index.ts]"]
  node_searchprovider["Public search provider"]
  node_demodiscovery["Demo discovery provider"]
  node_enrichment["Contact enrichment<br/>[index.ts]"]
  node_hunter["Hunter email provider"]
  node_website["Company website provider"]
  node_demoenrich["Demo enrichment provider"]
  node_export["CSV and XLSX generation<br/>[index.ts]"]
end

node_browser(("Research user"))
node_redis[("Redis")]
node_postgres[("PostgreSQL")]
node_serpapi["SerpApi search"]
node_hunterservice["Hunter.io"]
node_download["Downloaded file"]

node_browser -->|"uses"| node_web
node_web -->|"requests"| node_webapi
node_web -->|"displays"| node_webcomponents
node_web -->|"displays"| node_webresults
node_webapi -->|"calls REST"| node_server
node_server -->|"registers routes"| node_researchroutes
node_server -->|"registers routes"| node_leadroutes
node_server -->|"registers routes"| node_exportroutes
node_researchroutes -->|"enqueues job"| node_queue
node_researchroutes -->|"reads and writes"| node_database
node_queue -->|"stores jobs"| node_redis
node_worker -->|"consumes jobs"| node_queue
node_worker -->|"runs job"| node_pipeline
node_pipeline -->|"selects providers"| node_discovery
node_pipeline -->|"searches candidates"| node_searchprovider
node_pipeline -->|"searches candidates"| node_demodiscovery
node_searchprovider -.->|"queries search"| node_serpapi
node_pipeline -->|"scores and deduplicates"| node_verification
node_pipeline -->|"selects providers"| node_enrichment
node_pipeline -->|"enriches candidate"| node_website
node_pipeline -.->|"enriches candidate"| node_hunter
node_pipeline -->|"enriches candidate"| node_demoenrich
node_hunter -.->|"looks up email"| node_hunterservice
node_pipeline -->|"persists lead"| node_persist
node_persist -->|"writes leads"| node_database
node_database -->|"stores records"| node_postgres
node_pipeline -->|"updates job"| node_database
node_pipeline -->|"publishes progress"| node_queue
node_queue -->|"publishes events"| node_redis
node_researchroutes -->|"subscribes to progress"| node_redis
node_researchroutes -->|"streams SSE"| node_webapi
node_webapi -->|"delivers progress"| node_webcomponents
node_leadroutes -->|"queries leads"| node_database
node_exportroutes -->|"loads results"| node_database
node_exportroutes -->|"builds file"| node_export
node_export -->|"returns CSV or XLSX"| node_download
node_exportroutes -->|"returns export"| node_webapi
node_webapi -->|"delivers results"| node_webresults

click node_web "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/web/app/page.tsx"
click node_webcomponents "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/web/components/ProgressPanel.tsx"
click node_webresults "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/web/components/ResultsTable.tsx"
click node_webapi "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/web/lib/api.ts"
click node_server "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/api/src/server.ts"
click node_researchroutes "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/api/src/routes/research.ts"
click node_leadroutes "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/api/src/routes/leads.ts"
click node_exportroutes "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/api/src/routes/export.ts"
click node_queue "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/queue/src/index.ts"
click node_worker "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/worker/src/index.ts"
click node_pipeline "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/worker/src/pipeline/runResearchJob.ts"
click node_discovery "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/discovery/src/index.ts"
click node_searchprovider "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/discovery/src/providers/searchApiDiscoveryProvider.ts"
click node_demodiscovery "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/discovery/src/providers/demoDiscoveryProvider.ts"
click node_verification "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/verification/src/index.ts"
click node_enrichment "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/enrichment/src/index.ts"
click node_hunter "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/enrichment/src/providers/hunterEmailProvider.ts"
click node_website "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/enrichment/src/providers/companyWebsiteProvider.ts"
click node_demoenrich "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/enrichment/src/providers/demoEnrichmentProvider.ts"
click node_persist "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/worker/src/pipeline/persist.ts"
click node_database "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/database/src/index.ts"
click node_export "https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/export/src/index.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_web,node_webcomponents,node_webresults,node_webapi,node_browser toneBlue
class node_server,node_researchroutes,node_leadroutes,node_exportroutes,node_queue,node_redis,node_postgres toneAmber
class node_worker,node_pipeline,node_verification,node_persist,node_database,node_serpapi toneMint
class node_discovery,node_searchprovider,node_demodiscovery,node_enrichment,node_hunter,node_website,node_demoenrich,node_export toneRose
class node_hunterservice,node_download toneIndigo
```

## Components

### Browser experience

- [Search and results UI](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/web/app/page.tsx): `apps/web/app/page.tsx`
- [Progress, filters, details](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/web/components/ProgressPanel.tsx): `apps/web/components/ProgressPanel.tsx`
- [Results and exports UI](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/web/components/ResultsTable.tsx): `apps/web/components/ResultsTable.tsx`
- [API client](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/web/lib/api.ts): `apps/web/lib/api.ts`

### API and jobs

- [Fastify server](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/api/src/server.ts): `apps/api/src/server.ts`
- [Research jobs and SSE](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/api/src/routes/research.ts): `apps/api/src/routes/research.ts`
- [Lead queries](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/api/src/routes/leads.ts): `apps/api/src/routes/leads.ts`
- [Export endpoint](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/api/src/routes/export.ts): `apps/api/src/routes/export.ts`
- [Job queue and events](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/queue/src/index.ts): `packages/queue/src/index.ts`

### Research pipeline

- [Queue worker](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/worker/src/index.ts): `apps/worker/src/index.ts`
- [Research orchestration](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/worker/src/pipeline/runResearchJob.ts): `apps/worker/src/pipeline/runResearchJob.ts`
- [Ranking and dedup](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/verification/src/index.ts): `packages/verification/src/index.ts`
- [Lead persistence](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/apps/worker/src/pipeline/persist.ts): `apps/worker/src/pipeline/persist.ts`
- [Postgres data](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/database/src/index.ts): `packages/database/src/index.ts`

### Providers and exports

- [Candidate discovery](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/discovery/src/index.ts): `packages/discovery/src/index.ts`
- [Public search provider](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/discovery/src/providers/searchApiDiscoveryProvider.ts): `packages/discovery/src/providers/searchApiDiscoveryProvider.ts`
- [Demo discovery provider](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/discovery/src/providers/demoDiscoveryProvider.ts): `packages/discovery/src/providers/demoDiscoveryProvider.ts`
- [Contact enrichment](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/enrichment/src/index.ts): `packages/enrichment/src/index.ts`
- [Hunter email provider](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/enrichment/src/providers/hunterEmailProvider.ts): `packages/enrichment/src/providers/hunterEmailProvider.ts`
- [Company website provider](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/enrichment/src/providers/companyWebsiteProvider.ts): `packages/enrichment/src/providers/companyWebsiteProvider.ts`
- [Demo enrichment provider](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/enrichment/src/providers/demoEnrichmentProvider.ts): `packages/enrichment/src/providers/demoEnrichmentProvider.ts`
- [CSV and XLSX generation](https://github.com/ashutosh-kumar-rao/linkedin-crawler/blob/main/packages/export/src/index.ts): `packages/export/src/index.ts`

### Other components

- **Research user**
- **Redis**
- **PostgreSQL**
- **SerpApi search**
- **Hunter.io**
- **Downloaded file**

## Connections

- Research user → Search and results UI: uses
- Search and results UI → API client: requests
- Search and results UI → Progress, filters, details: displays
- Search and results UI → Results and exports UI: displays
- API client → Fastify server: calls REST
- Fastify server → Research jobs and SSE: registers routes
- Fastify server → Lead queries: registers routes
- Fastify server → Export endpoint: registers routes
- Research jobs and SSE → Job queue and events: enqueues job
- Research jobs and SSE → Postgres data: reads and writes
- Job queue and events → Redis: stores jobs
- Queue worker → Job queue and events: consumes jobs
- Queue worker → Research orchestration: runs job
- Research orchestration → Candidate discovery: selects providers
- Research orchestration → Public search provider: searches candidates
- Research orchestration → Demo discovery provider: searches candidates
- Public search provider → SerpApi search: queries search
- Research orchestration → Ranking and dedup: scores and deduplicates
- Research orchestration → Contact enrichment: selects providers
- Research orchestration → Company website provider: enriches candidate
- Research orchestration → Hunter email provider: enriches candidate
- Research orchestration → Demo enrichment provider: enriches candidate
- Hunter email provider → Hunter.io: looks up email
- Research orchestration → Lead persistence: persists lead
- Lead persistence → Postgres data: writes leads
- Postgres data → PostgreSQL: stores records
- Research orchestration → Postgres data: updates job
- Research orchestration → Job queue and events: publishes progress
- Job queue and events → Redis: publishes events
- Research jobs and SSE → Redis: subscribes to progress
- Research jobs and SSE → API client: streams SSE
- API client → Progress, filters, details: delivers progress
- Lead queries → Postgres data: queries leads
- Export endpoint → Postgres data: loads results
- Export endpoint → CSV and XLSX generation: builds file
- CSV and XLSX generation → Downloaded file: returns CSV or XLSX
- Export endpoint → API client: returns export
- API client → Results and exports UI: delivers results