# ergoncugler/web-scraping-telegram architecture

> Architecture diagram of the GitHub repository ergoncugler/web-scraping-telegram, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ergoncugler/web-scraping-telegram
- Repository: https://github.com/ergoncugler/web-scraping-telegram

## Overview

##### Project Type and Purpose
- This is an open-source, Python-based data-pipeline tool whose primary goal is to scrape messages from Telegram channels/groups, store them in Apache Parquet files, and then run a suite of post-processing/analysis scripts to produce combined datasets, monthly summaries, samples, keyword-filtered exports, and link analytics.
- There is no persistent web or mobile front-end; instead the “UI” is a Jupyter/Colab notebook (`TelegramScrap_A_comprehensive_tool_for_scraping_Telegram_data.ipynb`) plus a set of standalone helper scripts.

##### High-Level File Structure
- `TelegramScrap_A_comprehensive_tool_for_scraping_Telegram_data.ipynb` – main ingestion workflow using Telethon (async Telegram API client).
- `combine_scraped_parquet_files.py` – dedupe & merge multiple Parquet files.
- `generate_groups_month_summary.py` – build per-group, per-month summary tables.
- `scrape_and_filter_by_keywords_from_parquet_to_excel.py` – filter by keywords, add flags, export to Excel.
- `snowballing_scrape_telegram_links_from_data.py` – extract/normalize Telegram links from stored data and export counts.
- `README.md` – describes setup (credentials, parameters), usage tips, and research applications.

##### Key Components
A. Ingestion Layer
- Telethon Client (async Python code inside the notebook)
- Parameterization module (user-supplied credentials + scraping parameters)
- Execution environment (Google Colab or local Python interpreter)
- External Service: Telegram API endpoints
B. Storage Layer
- Raw data storage: Apache Parquet files
- Checkpoint/backups every 1,000 messages
C. Processing Layer (Post-Processing Scripts)
1. combine_scraped_parquet_files.py
2. generate_groups_month_summary.py
3. sample_data_from_parquet_to_excel.py
4. scrape_and_filter_by_keywords_from_parquet_to_excel.py
5. snowballing_scrape_telegram_links_from_data.py
D. Output/Delivery Layer
- Final combined Parquet files (for further analysis)
- Excel reports (samples, summaries, filters, link counts)
E. Optional External Consumers
- Data scientists/researchers using pandas, Excel, BI tools

##### Relationships & Data Flow
1. User populates credentials & parameters → notebook kicks off Telethon
2. Notebook ⇄ Telegram API → fetch message streams → write `.parquet` files
3. For every N messages scraped → backup Parquet snapshot
4. Post-processing scripts read one or more Parquet files → process → write new Parquet or Excel
5. Final outputs consumed by researchers or ingested into downstream analytics (e.g., BI dashboards)

##### Architectural Patterns & Principles
- Pipeline (discrete stages: ingestion → storage → processing → reporting)
- Modularity (each analysis step is its own script, enabling reuse & extension)
- Asynchronous API access (Telethon’s async calls to Telegram)
- Batch processing (Parquet for high-volume, columnar efficiency)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Ingestion Layer
    User["User / Credentials"]:::external
    Param["Parameterization\n(Credentials & Parameters)"]:::ingestion
    Execute["Execution Env\n(Colab / Local)"]:::external
    Telethon["Telethon Ingestor\n(Notebook)"]:::ingestion
    TelegramAPI["Telegram API\nendpoints"]:::external

    %% Storage Layer
    Parquet["Parquet Store\n(.parquet files)"]:::storage

    %% Processing Layer
    subgraph "Processing Layer"
        direction TB
        Combine["Combine Script"]:::processing
        Summary["Monthly Summary Script"]:::processing
        Sample["Sampling Script"]:::processing
        Filter["Keyword Filter Script"]:::processing
        LinkAnal["Link Analytics Script"]:::processing
    end

    %% Output Layer
    subgraph "Output Layer"
        direction TB
        CombinedParquet["Combined Parquet"]:::storage
        SummaryExcel["Summary Excel"]:::output
        SampleExcel["Sample Excel"]:::output
        FilterExcel["Filtered Excel"]:::output
        LinkExcel["Link Count Excel"]:::output
    end

    %% External Consumers
    Researchers["Researchers / BI Tools"]:::external

    %% Connections
    User -->|"provides credentials"| Param
    Param -->|"configures"| Telethon
    User -->|"initiates"| Execute
    Execute -->|"hosts notebook"| Telethon
    Telethon -->|"uses api_id/api_hash"| TelegramAPI
    Telethon -->|"write .parquet"| Parquet
    Parquet -->|"read .parquet"| Combine
    Parquet -->|"read .parquet"| Summary
    Parquet -->|"read .parquet"| Sample
    Parquet -->|"read .parquet"| Filter
    Parquet -->|"read .parquet"| LinkAnal
    Combine -->|"write .parquet"| CombinedParquet
    Summary -->|"export .xlsx"| SummaryExcel
    Sample -->|"export .xlsx"| SampleExcel
    Filter -->|"export .xlsx"| FilterExcel
    LinkAnal -->|"export .xlsx"| LinkExcel
    CombinedParquet -->|"used by"| Researchers
    SummaryExcel -->|"used by"| Researchers
    SampleExcel -->|"used by"| Researchers
    FilterExcel -->|"used by"| Researchers
    LinkExcel -->|"used by"| Researchers

    %% Click Events
    click Telethon "https://github.com/ergoncugler/web-scraping-telegram/blob/main/TelegramScrap_A_comprehensive_tool_for_scraping_Telegram_data.ipynb"
    click Param "https://github.com/ergoncugler/web-scraping-telegram/blob/main/README.md"
    click Combine "https://github.com/ergoncugler/web-scraping-telegram/blob/main/combine_scraped_parquet_files.py"
    click Summary "https://github.com/ergoncugler/web-scraping-telegram/blob/main/generate_groups_month_summary.py"
    click Sample "https://github.com/ergoncugler/web-scraping-telegram/blob/main/sample_data_from_parquet_to_excel.py"
    click Filter "https://github.com/ergoncugler/web-scraping-telegram/blob/main/scrape_and_filter_by_keywords_from_parquet_to_excel.py"
    click LinkAnal "https://github.com/ergoncugler/web-scraping-telegram/blob/main/snowballing_scrape_telegram_links_from_data.py"

    %% Styles
    classDef ingestion fill:#CCE5FF,stroke:#333,stroke-width:1px
    classDef storage fill:#CCFFCC,stroke:#333,stroke-width:1px
    classDef processing fill:#FFD699,stroke:#333,stroke-width:1px
    classDef output fill:#E5CCFF,stroke:#333,stroke-width:1px
    classDef external fill:#E0E0E0,stroke:#333,stroke-width:1px
```
