# joeyism/linkedin_scraper architecture

> Architecture diagram of the GitHub repository joeyism/linkedin_scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/joeyism/linkedin_scraper
- Repository: https://github.com/joeyism/linkedin_scraper

## Overview

LinkedIn Scraper is an async Playwright library for collecting LinkedIn profiles, company pages and posts, and job listings. Its principal workflow is to establish or reuse an authenticated browser session, navigate LinkedIn pages, extract structured records, and return Pydantic models; job search first discovers job URLs. Scrapers share browser-page utilities and progress callbacks. The README also documents manual and credential login, cookie/session reuse, and progress reporting. Some scraper implementations are only partially excerpted, so edges to models and shared layers are limited to relationships visible in the excerpts or README.

Main entry points: [`linkedin_scraper/scrapers/base.py`](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/base.py) (Base scraper), [`linkedin_scraper/scrapers/company_posts.py`](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/company_posts.py) (Company posts scraper), [`linkedin_scraper/scrapers/person.py`](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/person.py) (Profile scraper), [`linkedin_scraper/scrapers/company.py`](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/company.py) (Company scraper)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_session["Browser and authentication"]
  node_browser["Browser manager<br/>[browser.py]"]
  node_auth["Authentication<br/>[auth.py]"]
end

subgraph group_scraping["Scraping workflows"]
  node_person["Profile scraper<br/>[person.py]"]
  node_company["Company scraper<br/>[company.py]"]
  node_posts["Company posts scraper<br/>[company_posts.py]"]
  node_jobsearch["Job search scraper<br/>[job_search.py]"]
  node_job["Job scraper<br/>[job.py]"]
end

subgraph group_data["Structured data"]
  node_personmodel["Person records<br/>[person.py]"]
  node_companymodel["Company records<br/>[company.py]"]
  node_postmodel["Post records<br/>[post.py]"]
  node_jobmodel["Job records<br/>[job.py]"]
end

subgraph group_support["Shared runtime support"]
  node_base["Base scraper<br/>[base.py]"]
  node_utils["Browser utilities<br/>[utils.py]"]
  node_exceptions["Scraper exceptions<br/>[exceptions.py]"]
  node_callbacks["Progress callbacks<br/>[callbacks.py]"]
end

node_user(("Library user"))
node_linkedin(("LinkedIn"))

node_user -->|"opens session"| node_browser
node_browser -->|"loads session"| node_auth
node_auth -->|"authenticates"| node_linkedin
node_user -->|"requests profile"| node_person
node_user -->|"requests company"| node_company
node_user -->|"requests posts"| node_posts
node_user -->|"searches jobs"| node_jobsearch
node_user -->|"requests job"| node_job
node_person -.->|"navigates and extracts"| node_linkedin
node_company -.->|"navigates and extracts"| node_linkedin
node_posts -->|"navigates and extracts"| node_linkedin
node_jobsearch -->|"searches listings"| node_linkedin
node_job -->|"navigates and extracts"| node_linkedin
node_person -->|"uses helpers"| node_base
node_company -->|"uses helpers"| node_base
node_posts -->|"uses helpers"| node_base
node_jobsearch -->|"uses helpers"| node_base
node_job -->|"uses helpers"| node_base
node_base -->|"delegates helpers"| node_utils
node_base -->|"checks login"| node_auth
node_base -->|"raises errors"| node_exceptions
node_auth -->|"raises errors"| node_exceptions
node_person -->|"returns records"| node_personmodel
node_company -->|"returns records"| node_companymodel
node_posts -->|"creates records"| node_postmodel
node_job -->|"creates records"| node_jobmodel
node_base -->|"reports progress"| node_callbacks
node_jobsearch -->|"reports progress"| node_callbacks
node_posts -->|"reports progress"| node_callbacks

click node_browser "https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/core/browser.py"
click node_auth "https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/core/auth.py"
click node_person "https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/person.py"
click node_company "https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/company.py"
click node_posts "https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/company_posts.py"
click node_jobsearch "https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/job_search.py"
click node_job "https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/job.py"
click node_base "https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/base.py"
click node_utils "https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/core/utils.py"
click node_exceptions "https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/core/exceptions.py"
click node_callbacks "https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/callbacks.py"
click node_personmodel "https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/models/person.py"
click node_companymodel "https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/models/company.py"
click node_postmodel "https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/models/post.py"
click node_jobmodel "https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/models/job.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_browser,node_auth,node_user toneBlue
class node_person,node_company,node_posts,node_jobsearch,node_job toneAmber
class node_personmodel,node_companymodel,node_postmodel,node_jobmodel toneMint
class node_base,node_utils,node_exceptions,node_callbacks toneRose
class node_linkedin toneIndigo
```

## Components

### Browser and authentication

- [Browser manager](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/core/browser.py): `linkedin_scraper/core/browser.py`
- [Authentication](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/core/auth.py): `linkedin_scraper/core/auth.py`

### Scraping workflows

- [Profile scraper](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/person.py): `linkedin_scraper/scrapers/person.py`
- [Company scraper](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/company.py): `linkedin_scraper/scrapers/company.py`
- [Company posts scraper](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/company_posts.py): `linkedin_scraper/scrapers/company_posts.py`
- [Job search scraper](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/job_search.py): `linkedin_scraper/scrapers/job_search.py`
- [Job scraper](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/job.py): `linkedin_scraper/scrapers/job.py`

### Structured data

- [Person records](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/models/person.py): `linkedin_scraper/models/person.py`
- [Company records](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/models/company.py): `linkedin_scraper/models/company.py`
- [Post records](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/models/post.py): `linkedin_scraper/models/post.py`
- [Job records](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/models/job.py): `linkedin_scraper/models/job.py`

### Shared runtime support

- [Base scraper](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/scrapers/base.py): `linkedin_scraper/scrapers/base.py`
- [Browser utilities](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/core/utils.py): `linkedin_scraper/core/utils.py`
- [Scraper exceptions](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/core/exceptions.py): `linkedin_scraper/core/exceptions.py`
- [Progress callbacks](https://github.com/joeyism/linkedin_scraper/blob/master/linkedin_scraper/callbacks.py): `linkedin_scraper/callbacks.py`

### Other components

- **Library user**
- **LinkedIn**

## Connections

- Library user → Browser manager: opens session (evidence: `README.md`)
- Browser manager → Authentication: loads session (evidence: `README.md`)
- Authentication → LinkedIn: authenticates (evidence: `linkedin_scraper/core/auth.py`)
- Library user → Profile scraper: requests profile (evidence: `README.md`)
- Library user → Company scraper: requests company (evidence: `README.md`)
- Library user → Company posts scraper: requests posts (evidence: `README.md`)
- Library user → Job search scraper: searches jobs (evidence: `README.md`)
- Library user → Job scraper: requests job (evidence: `README.md`)
- Profile scraper → LinkedIn: navigates and extracts
- Company scraper → LinkedIn: navigates and extracts
- Company posts scraper → LinkedIn: navigates and extracts (evidence: `linkedin_scraper/scrapers/company_posts.py`)
- Job search scraper → LinkedIn: searches listings (evidence: `linkedin_scraper/scrapers/job_search.py`)
- Job scraper → LinkedIn: navigates and extracts (evidence: `linkedin_scraper/scrapers/job.py`)
- Profile scraper → Base scraper: uses helpers (evidence: `linkedin_scraper/scrapers/person.py`)
- Company scraper → Base scraper: uses helpers (evidence: `linkedin_scraper/scrapers/company.py`)
- Company posts scraper → Base scraper: uses helpers (evidence: `linkedin_scraper/scrapers/company_posts.py`)
- Job search scraper → Base scraper: uses helpers (evidence: `linkedin_scraper/scrapers/job_search.py`)
- Job scraper → Base scraper: uses helpers (evidence: `linkedin_scraper/scrapers/job.py`)
- Base scraper → Browser utilities: delegates helpers (evidence: `linkedin_scraper/scrapers/base.py`)
- Base scraper → Authentication: checks login (evidence: `linkedin_scraper/scrapers/base.py`)
- Base scraper → Scraper exceptions: raises errors (evidence: `linkedin_scraper/scrapers/base.py`)
- Authentication → Scraper exceptions: raises errors (evidence: `linkedin_scraper/core/auth.py`)
- Profile scraper → Person records: returns records (evidence: `linkedin_scraper/scrapers/person.py`)
- Company scraper → Company records: returns records (evidence: `linkedin_scraper/scrapers/company.py`)
- Company posts scraper → Post records: creates records (evidence: `linkedin_scraper/scrapers/company_posts.py`)
- Job scraper → Job records: creates records (evidence: `linkedin_scraper/scrapers/job.py`)
- Base scraper → Progress callbacks: reports progress (evidence: `linkedin_scraper/scrapers/base.py`)
- Job search scraper → Progress callbacks: reports progress (evidence: `linkedin_scraper/scrapers/job_search.py`)
- Company posts scraper → Progress callbacks: reports progress (evidence: `linkedin_scraper/scrapers/company_posts.py`)
