# arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing architecture

> Architecture diagram of the GitHub repository arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing
- Repository: https://github.com/arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing

## Overview

##### Project type and purpose
- This is a full-stack Python toolkit combining:
- A FastAPI web UI (app.py + Jinja2 templates)
- Standalone CLI scripts (scrapName.py, emailFilter.py, automateEmailing.py, etc.)
Purpose: Scrape authors from PubMed, validate email deliverability, and send personalized outreach at scale.

##### File structure
- Top-level scripts:
- app.py → FastAPI server entrypoint
- scrapName.py → PubMed scraper (NCBI Entrez API)
- emailFilter.py → Email validation pipeline (syntax, MX, SMTP probe)
- automateEmailing.py → Bulk mailing CLI
- for_automate_authentic_email_google_yahoo_office.py → Provider-specific SMTP helper
- universal.py → Shared helper functions (e.g., CSV I/O, templating)
- Directories:
- templates/ → Jinja2 HTML for web UI + example email templates
- fancyMails/, newSampleTemplates/ → Example HTML email templates
- Configuration & examples:
- requirements.txt, .env (uncommitted), try.csv / tryfilter.csv

##### README insights
- Tech stack: Python 3.8+, FastAPI, Jinja2, uvicorn, requests, dnspython, pydantic
- Workflow stages defined: Discovery → Validation → Delivery
- Two interfaces: Web UI & CLI
- Data format: CSV in/out, HTML templates with `{placeholders}`

## Diagram (Mermaid)

```mermaid
graph TB
    %% UI Layer
    subgraph "UI Layer"
        Browser["User Browser"]:::external
        CLI["CLI Terminal"]:::cli
    end

    %% Application Layer
    subgraph "Application Layer"
        subgraph "Web UI (FastAPI)"
            FastAPI["FastAPI Server"]:::web
            FastAPI --> FastAPIScrape
            FastAPI --> FastAPIFilter
            FastAPI --> FastAPISend
            FastAPIScrape["/email-scraper Endpoint"]:::web
            FastAPIFilter["/email-filter Endpoint"]:::web
            FastAPISend["/email-sender Endpoint"]:::web
        end
        subgraph "CLI Tools"
            Scrap["PubMed Scraper CLI"]:::cli
            Filter["Email Validation CLI"]:::cli
            Sender["Bulk Mailing CLI"]:::cli
        end
        Shared["Shared Business Logic (universal.py)"]:::logic
    end

    %% Data Layer
    subgraph "Data Layer"
        InputCSV["Input CSV (try.csv)"]:::data
        FilteredCSV["Filtered CSV (tryfilter.csv)"]:::data
        Templates["Web UI Templates (templates/)"]:::data
        subgraph "Email Templates"
            Fancy["FancyMails/ examples"]:::data
            Sample["newSampleTemplates/ examples"]:::data
        end
        README["README.md"]:::data
        ProcessNotes["webscrap and email automation Process.txt"]:::data
    end

    %% External Services
    subgraph "External Services"
        PubMedAPI["NCBI Entrez API"]:::external
        DNS["DNS/MX Resolver"]:::external
        SMTP["SMTP Servers"]:::external
    end

    %% Connections
    Browser -->|"HTTP Request"| FastAPI
    CLI --> Scrap
    CLI --> Filter
    CLI --> Sender

    FastAPI -->|"calls"| Shared
    Scrap -->|"uses"| Shared
    Filter -->|"uses"| Shared
    Sender -->|"uses"| Shared

    Shared -->|"reads/writes"| InputCSV
    Shared -->|"reads/writes"| FilteredCSV
    Shared -->|"renders/reads"| Templates
    Shared -->|"renders/reads"| Fancy
    Shared -->|"renders/reads"| Sample

    Shared -->|"HTTP REST"| PubMedAPI
    Shared -->|"DNS Lookup"| DNS
    Shared -->|"SMTP Send"| SMTP

    %% Click Events
    click FastAPI "https://github.com/arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing/blob/main/app.py"
    click Scrap "https://github.com/arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing/blob/main/scrapName.py"
    click Filter "https://github.com/arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing/blob/main/emailFilter.py"
    click Sender "https://github.com/arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing/blob/main/automateEmailing.py"
    click Shared "https://github.com/arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing/blob/main/universal.py"
    click Templates "https://github.com/arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing/tree/main/templates/"
    click Fancy "https://github.com/arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing/tree/main/fancyMails/"
    click Sample "https://github.com/arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing/tree/main/newSampleTemplates/"
    click InputCSV "https://github.com/arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing/blob/main/try.csv"
    click FilteredCSV "https://github.com/arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing/blob/main/tryfilter.csv"
    click README "https://github.com/arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing/blob/main/README.md"
    click ProcessNotes "https://github.com/arupa444/web-scraping-info-of-authors-though-api-to-automate-the-mailing/blob/main/webscrap and email automation Process.txt"

    %% Styles
    classDef web fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef cli fill:#d4edda,stroke:#333,stroke-width:1px
    classDef logic fill:#fff3cd,stroke:#333,stroke-width:1px
    classDef data fill:#fff5e6,stroke:#333,stroke-width:1px
    classDef external fill:#f8d7da,stroke:#333,stroke-width:1px
```
