# oyelamin/twitter-scrapi architecture

> Architecture diagram of the GitHub repository oyelamin/twitter-scrapi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/oyelamin/twitter-scrapi
- Repository: https://github.com/oyelamin/twitter-scrapi

## Overview

##### Project type and purpose
- This is a Python-based web scraping tool (an open-source utility) named “TwitterScrapper.”
- Its goal is to extract Twitter profile data, tweets, and media without using the official Twitter API by routing requests through the Nitter service.

##### File structure
- Top-level files:
- README.md – project overview, installation, usage, and architecture notes
- main.py – likely contains the CLI entry point or test harness that instantiates and invokes the scraper
- No separate “src” or “lib” folders; all functional code lives in a Python module (twitter_scrapper) imported by main.py.
- Dependencies and setup rely on Selenium (via webdriver-manager), BeautifulSoup, asyncio, and ThreadPoolExecutor for concurrency.

##### README for additional insights
- Core components described:
- Selenium (headless Chrome) to fetch HTML via Nitter
- BeautifulSoup to parse and extract fields
- Async functions and ThreadPoolExecutor to process requests concurrently

a. Main components
1. User / CLI – initiates scraping commands via main.py
2. TwitterScrapper class
– Async layer (asyncio event loop + ThreadPoolExecutor)
– Selenium WebDriver Manager
3. Headless Browser (ChromeDriver)
4. Nitter service (external HTTP endpoint proxying Twitter)
5. BeautifulSoup parser
6. Data Model / Output (profile JSON, tweet lists, media URLs)

b. Relationships and interactions
1. User invokes main.py → calls TwitterScrapper methods (`search()`, `get_profile()`, etc.).
2. TwitterScrapper schedules tasks on the asyncio event loop; CPU-bound or blocking operations are off-loaded to ThreadPoolExecutor.
3. For each task:
- Selenium WebDriver opens Nitter URL (e.g., https://nitter.net/\<username\>)
- Headless Chrome fetches rendered HTML
- HTML is returned to scraper
- BeautifulSoup parses HTML to extract fields
4. Data is transformed into Python dicts/lists and returned to caller.

c. Architectural patterns and principles
- Asynchronous I/O with thread-pool offloading for non-blocking scraping
- Separation of concerns:
- Fetching/rendering (Selenium)
- Parsing (BeautifulSoup)
- Orchestration (asyncio + ThreadPoolExecutor)
- External service integration (Nitter) abstracts away reliance on Twitter’s official API.

d. Key technologies and libraries
- Python 3.x
- asyncio (event loop, `async`/`await`)
- concurrent.futures.ThreadPoolExecutor
- Selenium + webdriver-manager (managing ChromeDriver lifecycle)
- bs4 (BeautifulSoup)
- Nitter (external HTTP service)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Local Components"
        CLI["User / CLI"]:::local
        Orchestrator["TwitterScrapper Orchestrator<br>(asyncio + ThreadPoolExecutor)"]:::local
        Selenium["Selenium WebDriver Manager"]:::local
        Browser["Headless Browser<br>(ChromeDriver)"]:::local
        Parser["BeautifulSoup Parser"]:::parser
    end

    subgraph "External Services"
        Nitter["Nitter Service"]:::external
    end

    subgraph "Data Output"
        Data["Structured Data Output<br>(JSON dicts/lists)"]:::data
    end

    CLI -->|"invoke methods"| Orchestrator
    Orchestrator -->|"manage WebDriver"| Selenium
    Selenium -->|"spawn browser"| Browser
    Browser -->|"HTTP request (rendered URL)"| Nitter
    Nitter -->|"HTML response"| Browser
    Browser -->|"return HTML"| Orchestrator
    Orchestrator -->|"provide HTML"| Parser
    Parser -->|"Parsed JSON"| Data

    click CLI "https://github.com/oyelamin/twitter-scrapi/blob/main/main.py"
    click Orchestrator "https://github.com/oyelamin/twitter-scrapi/blob/main/main.py"
    click Selenium "https://github.com/oyelamin/twitter-scrapi/blob/main/main.py"
    click Parser "https://github.com/oyelamin/twitter-scrapi/blob/main/main.py"
    click README "https://github.com/oyelamin/twitter-scrapi/blob/main/README.md"

    classDef local fill:#9ff,stroke:#333,stroke-width:1px
    classDef external fill:#f96,stroke:#333,stroke-width:1px
    classDef parser fill:#ff9,stroke:#333,stroke-width:1px
    classDef data fill:#9f9,stroke:#333,stroke-width:1px
```
