How ismaelfi/scrape-linkedin-posts works
Project type and purpose
• This is a simple, single-script open-source tool (“LinkedIn Post Scraper”) written in Python.
• Purpose: Automate retrieval of a LinkedIn user’s recent posts (content, reactions, comments) via Selenium & BeautifulSoup, then export to CSV.
File structure
• Top level:
• scrape.py – main Python script
• README.md – project overview, setup, usage
• LICENSE – licensing terms
• No subdirectories or microservices—just one script with an external cookie file at runtime.
Key components
a. User / CLI
b. scrape.py (Python application)
c. Cookies file (Netscape-format text)
d. Selenium API (Python binding)
e. ChromeDriver (WebDriver executable)
f. Google Chrome browser (headless or headed)
g. LinkedIn web server (external service)
h. BeautifulSoup parser
i. Pandas DataFrame + CSV writer (local data store)
Relationships and data flow
1. User invokes python scrape.py --cookies your_linkedin_cookies.txt.
2. scrape.py loads cookies → initializes Selenium WebDriver.
3. Selenium API calls ChromeDriver → launches Chrome → authenticates with LinkedIn using cookies.
4. Chrome navigates to LinkedIn user’s activity page → fetches HTML.
5. scrape.py hands raw HTML to BeautifulSoup → extracts posts, reactions, comments.
6. Data passed to Pandas → DataFrame → CSV file written to disk.
Architectural patterns / design principles
• Pipeline pattern: Input → Fetch → Parse → Transform → Store
• Modular separation of concerns:
• Browser automation (Selenium & ChromeDriver)
• HTML parsing (BeautifulSoup)
• Data handling/export (Pandas)
• Single responsibility: each library does one role
Technologies / frameworks to annotate
• Python 3.x
• Selenium (Python binding)
• ChromeDriver + Google Chrome
• BeautifulSoup4
• Pandas
• CSV (local file system)
Essential details to capture
• Authentication via cookies file
• External dependency on a real browser + driver
• Sequential pipeline stages
• Clear separation between browser automation, parsing, and storage