How ashutosh-kumar-rao/linkedin-crawler works
Lead Research Engine accepts a search query, creates a research job, and hands it to a worker for discovery, relevance ranking, deduplication, enrichment, confidence scoring, and persistence. The browser can follow progress over SSE, browse and filter saved results, inspect lead details, and export CSV or XLSX. The README documents demo and optional search/email providers; sampled source confirms the job, pipeline, progress, and export wiring, while several UI, provider, and storage internals are represented only at their documented module paths.
Main entry points: runResearchJob.ts Research orchestration · api.ts API client · index.ts Postgres data · research.ts Research jobs and SSE
Components 28
Browser experience
- Search and results UI
apps/web/app/page.tsx - Progress, filters, details
apps/web/components/ProgressPanel.tsx - Results and exports UI
apps/web/components/ResultsTable.tsx - API client
apps/web/lib/api.ts
API and jobs
- Fastify server
apps/api/src/server.ts - Research jobs and SSE
apps/api/src/routes/research.ts - Lead queries
apps/api/src/routes/leads.ts - Export endpoint
apps/api/src/routes/export.ts - Job queue and events
packages/queue/src/index.ts
Research pipeline
- Queue worker
apps/worker/src/index.ts - Research orchestration
apps/worker/src/pipeline/runResearchJob.ts - Ranking and dedup
packages/verification/src/index.ts - Lead persistence
apps/worker/src/pipeline/persist.ts - Postgres data
packages/database/src/index.ts
Providers and exports
- Candidate discovery
packages/discovery/src/index.ts - Public search provider
packages/discovery/src/providers/searchApiDiscoveryProvider.ts - Demo discovery provider
packages/discovery/src/providers/demoDiscoveryProvider.ts - Contact enrichment
packages/enrichment/src/index.ts - Hunter email provider
packages/enrichment/src/providers/hunterEmailProvider.ts - Company website provider
packages/enrichment/src/providers/companyWebsiteProvider.ts - Demo enrichment provider
packages/enrichment/src/providers/demoEnrichmentProvider.ts - CSV and XLSX generation
packages/export/src/index.ts
Other components
- Research user
- Redis
- PostgreSQL
- SerpApi search
- Hunter.io
- Downloaded file
Connections 38
- Research user → Search and results UI: uses
- Search and results UI → API client: requests
- Search and results UI → Progress, filters, details: displays
- Search and results UI → Results and exports UI: displays
- API client → Fastify server: calls REST
- Fastify server → Research jobs and SSE: registers routes
- Fastify server → Lead queries: registers routes
- Fastify server → Export endpoint: registers routes
- Research jobs and SSE → Job queue and events: enqueues job
- Research jobs and SSE → Postgres data: reads and writes
- Job queue and events → Redis: stores jobs
- Queue worker → Job queue and events: consumes jobs
- Queue worker → Research orchestration: runs job
- Research orchestration → Candidate discovery: selects providers
- Research orchestration → Public search provider: searches candidates
- Research orchestration → Demo discovery provider: searches candidates
- Public search provider → SerpApi search: queries search
- Research orchestration → Ranking and dedup: scores and deduplicates
- Research orchestration → Contact enrichment: selects providers
- Research orchestration → Company website provider: enriches candidate
- Research orchestration → Hunter email provider: enriches candidate
- Research orchestration → Demo enrichment provider: enriches candidate
- Hunter email provider → Hunter.io: looks up email
- Research orchestration → Lead persistence: persists lead
- Lead persistence → Postgres data: writes leads
- Postgres data → PostgreSQL: stores records
- Research orchestration → Postgres data: updates job
- Research orchestration → Job queue and events: publishes progress
- Job queue and events → Redis: publishes events
- Research jobs and SSE → Redis: subscribes to progress
- Research jobs and SSE → API client: streams SSE
- API client → Progress, filters, details: delivers progress
- Lead queries → Postgres data: queries leads
- Export endpoint → Postgres data: loads results
- Export endpoint → CSV and XLSX generation: builds file
- CSV and XLSX generation → Downloaded file: returns CSV or XLSX
- Export endpoint → API client: returns export
- API client → Results and exports UI: delivers results