# hammad112/linkedin-scrapper architecture

> Architecture diagram of the GitHub repository hammad112/linkedin-scrapper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hammad112/linkedin-scrapper
- Repository: https://github.com/hammad112/linkedin-scrapper

## Overview

##### Project type and purpose
- This is a standalone Python-based data-pipeline tool (“LinkedIn Scraper”) whose goal is to programmatically navigate LinkedIn company pages via Selenium/Chrome, extract structured company data, then clean and merge that data into usable JSON outputs.

##### File structure
Top-level:
- Configuration/ – global settings (chrome_profile_path)
- main.py – orchestrates the end-to-end pipeline
- companies_list_space.csv – input list of target companies
- Scraping_Companies/ – raw JSON outputs for each scraped company
- clean_empty_fields/ – module that removes empty fields from JSON
- json_merge/ – module that merges cleaned JSON files into a consolidated dataset
- Utilis/ – shared utility functions (e.g., file I/O, logging)
- scrapper_linkedin/ – core Selenium scraping logic
- requirements.txt – external dependencies

##### README for additional insights
- Uses Selenium with an existing Chrome profile (no login logic inside the code)
- Leverages pandas and Python’s json module for data cleaning and merging
- Three custom modules to handle dynamic content, clean empty fields, and merge JSON

##### Elements
- Component labels: “Orchestrator (main.py),” “Scraper Service,” etc.
– Rectangles for code modules/services
– Clouds for external systems (LinkedIn, Chrome)

##### Detailed capture of architectural elements
- Configuration Loader feeding credentials or paths into Scraper Service.
- “requirements.txt” off to the side as the dependency manifest for the entire pipeline.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Pipeline Components"
        Orchestrator["Orchestrator\n(main.py)"]:::app
        Config["Configuration Loader\n(config.py)"]:::config
        Scraper["Scraper Service\n(selenium + ChromeDriver)"]:::scraper
        Cleaner["Cleaner Module\n(data_clean.py)"]:::cleaner
        Merger["Merger Module\n(company_details.py)"]:::merger
        Util["Utility Library\n(utils.py)"]:::util
        Requirements["requirements.txt"]:::deps
    end

    subgraph "Data Stores"
        DataCSV["/companies_list_space.csv/"]:::data
        RawStore["(Scraping_Companies/)\nRaw JSON Store"]:::data
        CleanedStore["(Cleaned JSON Store)"]:::data
        FinalOut["/Final Output\n(merged JSON or DataFrame)/"]:::data
    end

    subgraph "External Systems"
        LinkedIn["LinkedIn Website"]:::external
        ChromeProfile["Chrome Browser Profile"]:::external
    end

    Orchestrator -->|loads| Config
    Orchestrator -->|reads| DataCSV
    Orchestrator -->|invokes| Scraper
    Config -->|provides settings| Scraper
    DataCSV -->|companies list| Scraper
    Scraper -->|requests pages| LinkedIn
    Scraper -->|uses profile| ChromeProfile
    Scraper -->|writes JSON| RawStore
    Orchestrator -->|invokes| Cleaner
    RawStore -->|raw JSON| Cleaner
    Cleaner -->|produces cleaned JSON| CleanedStore
    Orchestrator -->|invokes| Merger
    CleanedStore -->|input JSON| Merger
    Merger -->|outputs merged data| FinalOut

    Util -.-> Scraper
    Util -.-> Cleaner
    Util -.-> Merger
    Requirements -.-> Orchestrator

    %% Click events
    click Orchestrator "https://github.com/hammad112/linkedin-scrapper/blob/main/main.py"
    click Config "https://github.com/hammad112/linkedin-scrapper/blob/main/Configuration/config.py"
    click Scraper "https://github.com/hammad112/linkedin-scrapper/blob/main/scrapper_linkedin/scraper.py"
    click RawStore "https://github.com/hammad112/linkedin-scrapper/tree/main/Scraping_Companies/"
    click Cleaner "https://github.com/hammad112/linkedin-scrapper/blob/main/clean_empty_fields/data_clean.py"
    click Merger "https://github.com/hammad112/linkedin-scrapper/blob/main/json_merge/company_details.py"
    click Util "https://github.com/hammad112/linkedin-scrapper/blob/main/Utilis/utils.py"
    click DataCSV "https://github.com/hammad112/linkedin-scrapper/blob/main/companies_list_space.csv"
    click Requirements "https://github.com/hammad112/linkedin-scrapper/blob/main/requirements.txt"

    %% Styles
    classDef app fill:#D6E9C6,stroke:#3C763D,color:#3C763D;
    classDef config fill:#FCF8E3,stroke:#8A6D3B,color:#8A6D3B;
    classDef scraper fill:#D9EDF7,stroke:#31708F,color:#31708F;
    classDef cleaner fill:#F2DEDE,stroke:#A94442,color:#A94442;
    classDef merger fill:#E8F0FE,stroke:#1967D2,color:#1967D2;
    classDef util fill:#F5F5F5,stroke:#333333,color:#333333,stroke-dasharray: 5 5;
    classDef data fill:#FFFFFF,stroke:#333333,color:#333333,stroke-dasharray: 2 2;
    classDef external fill:#F0F0F0,stroke:#999999,color:#666666;
    classDef deps fill:#FCF8E3,stroke:#8A6D3B,color:#8A6D3B,stroke-dasharray: 5 5;
```
