# fardinrehman/linkedin-research-data-collection-tool architecture

> Architecture diagram of the GitHub repository fardinrehman/linkedin-research-data-collection-tool, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fardinrehman/linkedin-research-data-collection-tool
- Repository: https://github.com/fardinrehman/linkedin-research-data-collection-tool

## Overview

The product is a LinkedIn research and data-collection application with a browser UI, a backend collection workflow, and a standalone browser simulation fallback. Users configure a search and fields, launch a job, monitor progress, inspect projected results and failures, then export data. The README documents simulation and authorized-provider adapters, dynamic projection, deduplication, SSE progress, and CSV/XLSX/JSON export. Several server internals are indexed but not excerpted; edges to those stages are shown only where the job-manager excerpt or README supports them.

Main entry points: [`client/src/App.jsx`](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/App.jsx) (Research workspace), [`server/src/services/jobManager.js`](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/services/jobManager.js) (Job manager), [`client/src/services/api.js`](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/services/api.js) (API and fallback client), [`server/src/server.js`](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/server.js) (HTTP server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["Research interface"]
  node_app["Research workspace<br/>[App.jsx]"]
  node_query["Search form"]
  node_selector["Field selector<br/>[FieldSelector.jsx]"]
  node_customfield["Custom field editor"]
  node_fieldmodel["Field presets<br/>[fields.js]"]
  node_streamhook["Job stream hook<br/>[useJobStream.js]"]
  node_statusui["Progress monitor<br/>[JobStatusCard.jsx]"]
  node_api_client["API and fallback client<br/>[api.js]"]
  node_browser_engine["Browser simulation<br/>[clientProvider.js]"]
end

subgraph group_api["Job API"]
  node_server["HTTP server<br/>[server.js]"]
  node_jobroutes["Job routes<br/>[jobRoutes.js]"]
  node_jobcontroller["Job controller<br/>[jobController.js]"]
  node_resultcontroller["Result controller"]
  node_exportcontroller["Export controller"]
end

subgraph group_collection["Collection processing"]
  node_jobmanager["Job manager<br/>[jobManager.js]"]
  node_normalize["Record normalization"]
  node_dedup["Composite deduplication"]
  node_jobstate[("In-memory job state<br/>[jobManager.js]")]
end

subgraph group_providers["Data adapters"]
  node_providerfactory["Provider selection<br/>[ProviderFactory.js]"]
  node_baseprovider["Provider contract"]
  node_mockprovider["Simulation adapter"]
  node_linkedinprovider["Authorized adapter"]
end

subgraph group_results["Results and export"]
  node_resultui["Dynamic results table"]
  node_failureui["Failed-record drawer"]
  node_exportui["Export toolbar<br/>[ExportToolbar.jsx]"]
  node_exportservice["Export generation<br/>[exportService.js]"]
end

node_researcher(("Researcher"))
node_externaldata{{"Authorized data service"}}

node_researcher -->|"uses"| node_app
node_app -->|"renders"| node_query
node_query -->|"renders"| node_selector
node_selector -->|"opens"| node_customfield
node_app -->|"loads presets"| node_fieldmodel
node_app -->|"creates and fetches jobs"| node_api_client
node_app -->|"tracks job"| node_streamhook
node_app -->|"renders status"| node_statusui
node_streamhook -->|"polls status"| node_api_client
node_api_client -->|"falls back to"| node_browser_engine
node_api_client -->|"calls HTTP API"| node_server
node_app -->|"renders results"| node_resultui
node_app -->|"renders diagnostics"| node_failureui
node_server -->|"mounts"| node_jobroutes
node_server -->|"mounts results"| node_resultcontroller
node_server -->|"mounts exports"| node_exportcontroller
node_jobroutes -->|"routes requests"| node_jobcontroller
node_jobcontroller -->|"creates and reads jobs"| node_jobmanager
node_jobcontroller -->|"lists providers"| node_providerfactory
node_jobmanager -->|"stores job state"| node_jobstate
node_jobmanager -->|"selects adapter"| node_providerfactory
node_jobmanager -->|"projects records"| node_normalize
node_jobmanager -->|"checks duplicates"| node_dedup
node_providerfactory -->|"selects simulation"| node_mockprovider
node_providerfactory -->|"selects authorized"| node_linkedinprovider
node_mockprovider -->|"implements contract"| node_baseprovider
node_linkedinprovider -->|"implements contract"| node_baseprovider
node_linkedinprovider -.->|"requests authorized data"| node_externaldata
node_resultcontroller -->|"reads job results"| node_jobmanager
node_exportcontroller -->|"reads job records"| node_jobmanager
node_exportcontroller -->|"generates files"| node_exportservice
node_resultui -->|"renders toolbar"| node_exportui
node_exportui -->|"requests export"| node_api_client
node_api_client -.->|"downloads via API"| node_exportservice

click node_app "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/App.jsx"
click node_query "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/components/search/SearchQueryForm.jsx"
click node_selector "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/components/search/FieldSelector.jsx"
click node_customfield "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/components/search/CustomFieldModal.jsx"
click node_fieldmodel "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/types/fields.js"
click node_streamhook "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/hooks/useJobStream.js"
click node_statusui "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/components/monitor/JobStatusCard.jsx"
click node_api_client "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/services/api.js"
click node_browser_engine "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/services/clientProvider.js"
click node_server "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/server.js"
click node_jobroutes "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/routes/jobRoutes.js"
click node_jobcontroller "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/controllers/jobController.js"
click node_resultcontroller "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/controllers/resultController.js"
click node_exportcontroller "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/controllers/exportController.js"
click node_jobmanager "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/services/jobManager.js"
click node_normalize "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/services/normalizationService.js"
click node_dedup "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/services/deduplicationService.js"
click node_providerfactory "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/providers/ProviderFactory.js"
click node_baseprovider "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/providers/BaseDataProvider.js"
click node_mockprovider "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/providers/MockDataProvider.js"
click node_linkedinprovider "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/providers/LinkedInAuthorizedProvider.js"
click node_resultui "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/components/results/DynamicDataTable.jsx"
click node_failureui "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/components/error/FailedRecordsDrawer.jsx"
click node_exportui "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/components/results/ExportToolbar.jsx"
click node_exportservice "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/services/exportService.js"
click node_jobstate "https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/services/jobManager.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_query,node_selector,node_customfield,node_fieldmodel,node_streamhook,node_statusui,node_api_client,node_browser_engine toneBlue
class node_server,node_jobroutes,node_jobcontroller,node_resultcontroller,node_exportcontroller toneAmber
class node_jobmanager,node_normalize,node_dedup,node_jobstate toneMint
class node_providerfactory,node_baseprovider,node_mockprovider,node_linkedinprovider toneRose
class node_resultui,node_failureui,node_exportui,node_exportservice,node_researcher,node_externaldata toneIndigo
```

## Components

### Research interface

- [Research workspace](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/App.jsx): `client/src/App.jsx`
- [Search form](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/components/search/SearchQueryForm.jsx): `client/src/components/search/SearchQueryForm.jsx`
- [Field selector](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/components/search/FieldSelector.jsx): `client/src/components/search/FieldSelector.jsx`
- [Custom field editor](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/components/search/CustomFieldModal.jsx): `client/src/components/search/CustomFieldModal.jsx`
- [Field presets](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/types/fields.js): `client/src/types/fields.js`
- [Job stream hook](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/hooks/useJobStream.js): `client/src/hooks/useJobStream.js`
- [Progress monitor](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/components/monitor/JobStatusCard.jsx): `client/src/components/monitor/JobStatusCard.jsx`
- [API and fallback client](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/services/api.js): `client/src/services/api.js`
- [Browser simulation](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/services/clientProvider.js): `client/src/services/clientProvider.js`

### Job API

- [HTTP server](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/server.js): `server/src/server.js`
- [Job routes](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/routes/jobRoutes.js): `server/src/routes/jobRoutes.js`
- [Job controller](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/controllers/jobController.js): `server/src/controllers/jobController.js`
- [Result controller](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/controllers/resultController.js): `server/src/controllers/resultController.js`
- [Export controller](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/controllers/exportController.js): `server/src/controllers/exportController.js`

### Collection processing

- [Job manager](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/services/jobManager.js): `server/src/services/jobManager.js`
- [Record normalization](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/services/normalizationService.js): `server/src/services/normalizationService.js`
- [Composite deduplication](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/services/deduplicationService.js): `server/src/services/deduplicationService.js`
- [In-memory job state](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/services/jobManager.js): `server/src/services/jobManager.js`

### Data adapters

- [Provider selection](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/providers/ProviderFactory.js): `server/src/providers/ProviderFactory.js`
- [Provider contract](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/providers/BaseDataProvider.js): `server/src/providers/BaseDataProvider.js`
- [Simulation adapter](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/providers/MockDataProvider.js): `server/src/providers/MockDataProvider.js`
- [Authorized adapter](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/providers/LinkedInAuthorizedProvider.js): `server/src/providers/LinkedInAuthorizedProvider.js`

### Results and export

- [Dynamic results table](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/components/results/DynamicDataTable.jsx): `client/src/components/results/DynamicDataTable.jsx`
- [Failed-record drawer](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/components/error/FailedRecordsDrawer.jsx): `client/src/components/error/FailedRecordsDrawer.jsx`
- [Export toolbar](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/client/src/components/results/ExportToolbar.jsx): `client/src/components/results/ExportToolbar.jsx`
- [Export generation](https://github.com/fardinrehman/linkedin-research-data-collection-tool/blob/main/server/src/services/exportService.js): `server/src/services/exportService.js`

### Other components

- **Researcher**
- **Authorized data service**

## Connections

- Researcher → Research workspace: uses (evidence: `README.md`)
- Research workspace → Search form: renders (evidence: `client/src/App.jsx`)
- Search form → Field selector: renders (evidence: `client/src/components/search/SearchQueryForm.jsx`)
- Field selector → Custom field editor: opens (evidence: `client/src/components/search/FieldSelector.jsx`)
- Research workspace → Field presets: loads presets (evidence: `client/src/App.jsx`)
- Research workspace → API and fallback client: creates and fetches jobs (evidence: `client/src/App.jsx`)
- Research workspace → Job stream hook: tracks job (evidence: `client/src/App.jsx`)
- Research workspace → Progress monitor: renders status (evidence: `client/src/App.jsx`)
- Job stream hook → API and fallback client: polls status (evidence: `client/src/hooks/useJobStream.js`)
- API and fallback client → Browser simulation: falls back to (evidence: `client/src/services/api.js`)
- API and fallback client → HTTP server: calls HTTP API (evidence: `client/src/services/api.js`)
- Research workspace → Dynamic results table: renders results (evidence: `client/src/App.jsx`)
- Research workspace → Failed-record drawer: renders diagnostics (evidence: `client/src/App.jsx`)
- HTTP server → Job routes: mounts (evidence: `server/src/server.js`)
- HTTP server → Result controller: mounts results (evidence: `server/src/server.js`)
- HTTP server → Export controller: mounts exports (evidence: `server/src/server.js`)
- Job routes → Job controller: routes requests (evidence: `server/src/routes/jobRoutes.js`)
- Job controller → Job manager: creates and reads jobs (evidence: `server/src/controllers/jobController.js`)
- Job controller → Provider selection: lists providers (evidence: `server/src/controllers/jobController.js`)
- Job manager → In-memory job state: stores job state (evidence: `server/src/services/jobManager.js`)
- Job manager → Provider selection: selects adapter (evidence: `server/src/services/jobManager.js`)
- Job manager → Record normalization: projects records (evidence: `server/src/services/jobManager.js`)
- Job manager → Composite deduplication: checks duplicates (evidence: `server/src/services/jobManager.js`)
- Provider selection → Simulation adapter: selects simulation (evidence: `server/src/providers/ProviderFactory.js`)
- Provider selection → Authorized adapter: selects authorized (evidence: `server/src/providers/ProviderFactory.js`)
- Simulation adapter → Provider contract: implements contract (evidence: `server/src/providers/MockDataProvider.js`)
- Authorized adapter → Provider contract: implements contract (evidence: `server/src/providers/LinkedInAuthorizedProvider.js`)
- Authorized adapter → Authorized data service: requests authorized data (evidence: `README.md`)
- Result controller → Job manager: reads job results (evidence: `server/src/controllers/resultController.js`)
- Export controller → Job manager: reads job records (evidence: `server/src/controllers/exportController.js`)
- Export controller → Export generation: generates files (evidence: `server/src/controllers/exportController.js`)
- Dynamic results table → Export toolbar: renders toolbar (evidence: `client/src/components/results/DynamicDataTable.jsx`)
- Export toolbar → API and fallback client: requests export (evidence: `client/src/components/results/ExportToolbar.jsx`)
- API and fallback client → Export generation: downloads via API (evidence: `README.md`)
