# robertaitch/telegram-scraper architecture

> Architecture diagram of the GitHub repository robertaitch/telegram-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/robertaitch/telegram-scraper
- Repository: https://github.com/robertaitch/telegram-scraper

## Overview

##### Project type and purpose
- This is a standalone, command-line Python scraper (not a web app or microservice).
- Its purpose: connect to Telegram via Telethon, continuously or on-demand scrape messages & media from one or more channels, store data in per-channel SQLite databases and file-system media directories, and export results to JSON/CSV.

##### File structure
- There is a single entrypoint: telegram-scraper.py
- Dependencies in requirements.txt: telethon (Telegram API client), aiohttp/asyncio (async network & concurrency)
- No separate frontend or backend directories—everything lives in one script.
- Database files and media directories are created dynamically under each channel’s folder.
- Exported files (CSV/JSON) are saved under the same per-channel folders.

##### Key insights from README
- Telethon-based message fetching
- Async event loop for concurrency: scraping loop, media download tasks, DB batch inserts, state-save timer
- SQLite DB connection pooling & batch operations
- Media storage pipeline with configurable parallelism
- Interactive CLI menu driving high-level commands
- Export streams for large datasets

a. User / CLI Interface
- The interactive menu (Add, Remove, Scrape, Continuous, Export, View, Quit)
b. Scraper Engine
- Telethon client session (async)
- Message fetcher (pagination, flood-control, continuous polling)
c. Media Download Pipeline
- Async download workers (max_concurrent_downloads)
- Retry logic with exponential backoff
- File-system writes to ./<channel>/media/
d. Data Storage Layer
- SQLite database per channel (connection pool, batch insert)
- Schema: messages table with indexed columns
- Periodic state save (every N messages)
e. Export Module
- Streaming CSV/JSON writer (memory-efficient) reading from DB
f. External Services & Resources
- Telegram API servers
- Local file system (for DBs, media, exports)

##### Relationships & data flow
- User CLI → Scraper Engine triggers Telethon calls
- Telethon client ↔ Telegram API (network calls)
- Scraper Engine → Data Storage (batch insert via pool)
- Scraper Engine → Media Pipeline → File System
- Export Module ↔ Data Storage → Exported CSV/JSON files
- State Manager → File System for progress checkpoints

##### Architectural patterns & principles
- Asynchronous I/O (asyncio event loop) for concurrency
- Batch processing for performance (batch_size)
- Connection pooling for DB efficiency
- Modular design: separate concerns (CLI, networking, storage, media, export)
- Resilience: retry with backoff, state snapshotting

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "User Interface"
        CLI["CLI (Add/Remove/Scrape/Continuous/Export/View)"]:::core
    end

    subgraph "Core Components"
        Engine["Scraper Engine\n(asyncio loop,\nTelethon client,\nmessage fetcher)"]:::core
        Media["Media Download Pipeline\n(async workers,\nretry/backoff)"]:::core
        Export["Export Module\n(streaming CSV/JSON)"]:::core
    end

    subgraph "Storage Layer"
        DB["SQLite DB per channel\n(connection pool,\nbatch inserts)"]:::storage
        FS["File System\n(channel folder:\n.db, media/, exports/)"]:::storage
    end

    ExternalService["Telegram API"]:::external

    subgraph "Execution Modes"
        OneTime["One-time Scrape"]:::core
        Continuous["Continuous Mode"]:::core
    end

    Dependencies["Dependencies\n(telethon, aiohttp/asyncio)"]:::core
    README["Documentation & Usage\n(architecture overview, CLI commands)"]:::core

    CLI -->|"commands"| Engine
    OneTime --> Engine
    Continuous --> Engine
    Engine -->|"get_messages()"| ExternalService
    Engine -->|"batch_insert()"| DB
    Engine -->|"download_file()"| Media
    Media -->|"write_media()"| FS
    Export -->|"read/query"| DB
    Export -->|"write_csv()/json()"| FS

    click CLI "https://github.com/robertaitch/telegram-scraper/blob/main/telegram-scraper.py"
    click Engine "https://github.com/robertaitch/telegram-scraper/blob/main/telegram-scraper.py"
    click Media "https://github.com/robertaitch/telegram-scraper/blob/main/telegram-scraper.py"
    click DB "https://github.com/robertaitch/telegram-scraper/blob/main/telegram-scraper.py"
    click Export "https://github.com/robertaitch/telegram-scraper/blob/main/telegram-scraper.py"
    click Dependencies "https://github.com/robertaitch/telegram-scraper/blob/main/requirements.txt"
    click README "https://github.com/robertaitch/telegram-scraper/blob/main/README.md"

    classDef core fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef storage fill:#90EE90,stroke:#000,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#000,stroke-width:1px
```
