How robertaitch/telegram-scraper works
Project type and purpose
• This is a standalone, command-line Python scraper (not a web app or microservice).
• Its purpose: connect to Telegram via Telethon, continuously or on-demand scrape messages & media from one or more channels, store data in per-channel SQLite databases and file-system media directories, and export results to JSON/CSV.
File structure
• There is a single entrypoint: telegram-scraper.py
• Dependencies in requirements.txt: telethon (Telegram API client), aiohttp/asyncio (async network & concurrency)
• No separate frontend or backend directories—everything lives in one script.
• Database files and media directories are created dynamically under each channel’s folder.
• Exported files (CSV/JSON) are saved under the same per-channel folders.
Key insights from README
• Telethon-based message fetching
• Async event loop for concurrency: scraping loop, media download tasks, DB batch inserts, state-save timer
• SQLite DB connection pooling & batch operations
• Media storage pipeline with configurable parallelism
• Interactive CLI menu driving high-level commands
• Export streams for large datasets
a. User / CLI Interface
• The interactive menu (Add, Remove, Scrape, Continuous, Export, View, Quit)
b. Scraper Engine
• Telethon client session (async)
• Message fetcher (pagination, flood-control, continuous polling)
c. Media Download Pipeline
• Async download workers (max_concurrent_downloads)
• Retry logic with exponential backoff
• File-system writes to ./<channel>/media/
d. Data Storage Layer
• SQLite database per channel (connection pool, batch insert)
• Schema: messages table with indexed columns
• Periodic state save (every N messages)
e. Export Module
• Streaming CSV/JSON writer (memory-efficient) reading from DB
f. External Services & Resources
• Telegram API servers
• Local file system (for DBs, media, exports)
Relationships & data flow
• User CLI → Scraper Engine triggers Telethon calls
• Telethon client ↔ Telegram API (network calls)
• Scraper Engine → Data Storage (batch insert via pool)
• Scraper Engine → Media Pipeline → File System
• Export Module ↔ Data Storage → Exported CSV/JSON files
• State Manager → File System for progress checkpoints
Architectural patterns & principles
• Asynchronous I/O (asyncio event loop) for concurrency
• Batch processing for performance (batch_size)
• Connection pooling for DB efficiency
• Modular design: separate concerns (CLI, networking, storage, media, export)
• Resilience: retry with backoff, state snapshotting