How hammad112/linkedin-scrapper works
Project type and purpose
• This is a standalone Python-based data-pipeline tool (“LinkedIn Scraper”) whose goal is to programmatically navigate LinkedIn company pages via Selenium/Chrome, extract structured company data, then clean and merge that data into usable JSON outputs.
File structure
Top-level:
• Configuration/ – global settings (chrome_profile_path)
• main.py – orchestrates the end-to-end pipeline
• companies_list_space.csv – input list of target companies
• Scraping_Companies/ – raw JSON outputs for each scraped company
• clean_empty_fields/ – module that removes empty fields from JSON
• json_merge/ – module that merges cleaned JSON files into a consolidated dataset
• Utilis/ – shared utility functions (e.g., file I/O, logging)
• scrapper_linkedin/ – core Selenium scraping logic
• requirements.txt – external dependencies
README for additional insights
• Uses Selenium with an existing Chrome profile (no login logic inside the code)
• Leverages pandas and Python’s json module for data cleaning and merging
• Three custom modules to handle dynamic content, clean empty fields, and merge JSON
Elements
• Component labels: “Orchestrator (main.py),” “Scraper Service,” etc.
– Rectangles for code modules/services
– Clouds for external systems (LinkedIn, Chrome)
Detailed capture of architectural elements
• Configuration Loader feeding credentials or paths into Scraper Service.
• “requirements.txt” off to the side as the dependency manifest for the entire pipeline.