How oyelamin/twitter-scrapi works
Project type and purpose
• This is a Python-based web scraping tool (an open-source utility) named “TwitterScrapper.”
• Its goal is to extract Twitter profile data, tweets, and media without using the official Twitter API by routing requests through the Nitter service.
File structure
• Top-level files:
• README.md – project overview, installation, usage, and architecture notes
• main.py – likely contains the CLI entry point or test harness that instantiates and invokes the scraper
• No separate “src” or “lib” folders; all functional code lives in a Python module (twitter_scrapper) imported by main.py.
• Dependencies and setup rely on Selenium (via webdriver-manager), BeautifulSoup, asyncio, and ThreadPoolExecutor for concurrency.
README for additional insights
• Core components described:
• Selenium (headless Chrome) to fetch HTML via Nitter
• BeautifulSoup to parse and extract fields
• Async functions and ThreadPoolExecutor to process requests concurrently
a. Main components
1. User / CLI – initiates scraping commands via main.py
2. TwitterScrapper class
– Async layer (asyncio event loop + ThreadPoolExecutor)
– Selenium WebDriver Manager
3. Headless Browser (ChromeDriver)
4. Nitter service (external HTTP endpoint proxying Twitter)
5. BeautifulSoup parser
6. Data Model / Output (profile JSON, tweet lists, media URLs)
b. Relationships and interactions
1. User invokes main.py → calls TwitterScrapper methods (search(), get_profile(), etc.).
2. TwitterScrapper schedules tasks on the asyncio event loop; CPU-bound or blocking operations are off-loaded to ThreadPoolExecutor.
3. For each task:
• Selenium WebDriver opens Nitter URL (e.g., https://nitter.net/\<username\>)
• Headless Chrome fetches rendered HTML
• HTML is returned to scraper
• BeautifulSoup parses HTML to extract fields
4. Data is transformed into Python dicts/lists and returned to caller.
c. Architectural patterns and principles
• Asynchronous I/O with thread-pool offloading for non-blocking scraping
• Separation of concerns:
• Fetching/rendering (Selenium)
• Parsing (BeautifulSoup)
• Orchestration (asyncio + ThreadPoolExecutor)
• External service integration (Nitter) abstracts away reliance on Twitter’s official API.
d. Key technologies and libraries
• Python 3.x
• asyncio (event loop, async/await)
• concurrent.futures.ThreadPoolExecutor
• Selenium + webdriver-manager (managing ChromeDriver lifecycle)
• bs4 (BeautifulSoup)
• Nitter (external HTTP service)