# ferru97/pypaperbot architecture

> Architecture diagram of the GitHub repository ferru97/pypaperbot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ferru97/pypaperbot
- Repository: https://github.com/ferru97/pypaperbot

## Overview

##### Project’s Main Purpose and Components
- The project is a CLI-driven automated downloader for scholarly papers.
- Its goal is to execute various queries or DOI-based requests to fetch papers and corresponding metadata.
- It interacts with external sources (e.g., Google Scholar, Crossref, SciHub, Annas Archive) and supports features like filtering (by year, journal, etc.) and proxy usage.

##### Break Down the Application into Core Modules
- Modules for External Data Access:
- Scholar.py: Handles queries to Google Scholar and processes search results.
- Crossref.py: Manages requests to Crossref to retrieve bibtex or metadata.
- Proxy.py & NetInfo.py: Manage network connections and proxy configurations, aiding in accessing blocked resources.
- Modules for Downloading and Parsing:
- Downloader.py: Central logic for downloading papers and associated content (PDFs, bibtex).
- HTMLparsers.py: Implements parsing of HTML content from external websites (e.g., Scholar pages).
- Modules for Data and Filtering:
- Paper.py: Likely encapsulates paper metadata and file naming conventions.
- PapersFilters.py: Provides filtering capabilities based on input criteria (year, citations, journal names).
- Utils.py: Contains helper utilities used across the application.

##### External Interactions
- Google Scholar (for search queries and citation-based retrieval).
- Crossref service (for metadata and bibtex).
- SciHub/SciDB/Annas Archive (for obtaining the PDF files).
- External systems to the relevant modules (Scholar.py, Crossref.py, Downloader.py).

##### A Data Flow/Workflow Layer
- Sequence:
1. User invokes PyPaperBot with CLI arguments.
2. __main__.py parses the input and calls appropriate module routines.
3. The Scholar module is queried if the input is search based, which upon receiving responses, passes data to filtering (via PapersFilters.py) and then to Downloader.py.
4. Similarly for DOI-based queries, Crossref.py and Downloader.py take the lead.
5. Utilities (Utils.py) and network management (NetInfo.py/Proxy.py) support all modules.

##### Architectural Patterns and Technologies
- Modular design: each functionality is encapsulated in its own module, making the system easier to maintain and extend.
- Use of dependency injection aspects – for example, proxy details passed from CLI to the network module.
- Mention that although not explicitly split into a frontend/backend, the application's clear separation of concerns (parsing, downloading, filtering) is its key architectural strength.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Input & Application Start
    subgraph "User Input & Application Start"
        CLI("Command-Line Interface (PyPaperBot/__main__.py)"):::cli
        Main("Main Application (__main__.py)"):::main
    end

    %% Internal Components
    subgraph "Internal Components"
        Scholar("Scholar.py - GoogleScholar Module"):::module
        Crossref("Crossref.py - Crossref Module"):::module
        Filter("PapersFilters.py - Data Filtering Module"):::module
        Downloader("Downloader.py - Downloader Module"):::module
        HTMLParser("HTMLparsers.py - HTML Parsing Module"):::module
        Paper("Paper.py - Paper Metadata Module"):::module
        Utils("Utils.py - Utility Functions"):::utility
        NetProxy("NetInfo.py & proxy.py - Network/Proxy Management"):::module
    end

    %% External Systems
    subgraph "External Systems"
        GS("Google Scholar"):::external
        CR("Crossref Service"):::external
        Archive("SciHub/SciDB/Annas Archive"):::external
    end

    %% Connections from User Start
    CLI -->|"CLI_args"| Main

    %% Main Application triggers internal modules
    Main -->|"Query(search)"| Scholar
    Main -->|"DOI_request"| Crossref
    Main -->|"Proxy_config"| NetProxy
    Main -->|"Helper_call"| Utils

    %% External interactions from internal modules
    Scholar -->|"send_search_query"| GS
    Scholar -->|"parsed_results"| Filter
    Crossref -->|"metadata_request"| CR

    %% Data flow towards downloading
    Filter -->|"filtered_data"| Downloader
    Crossref -->|"metadata"| Downloader
    Downloader -->|"download_request"| Archive

    %% Support via Network/Proxy Management (dashed lines)
    NetProxy -.->|"proxy"| Scholar
    NetProxy -.->|"proxy"| Crossref
    NetProxy -.->|"proxy"| Downloader

    %% Additional support from Utility Functions
    Utils -.->|"utilities"| Downloader

    %% Click Events
    click CLI "https://github.com/ferru97/pypaperbot/blob/master/PyPaperBot/__main__.py"
    click Main "https://github.com/ferru97/pypaperbot/blob/master/PyPaperBot/__main__.py"
    click Scholar "https://github.com/ferru97/pypaperbot/blob/master/PyPaperBot/Scholar.py"
    click Crossref "https://github.com/ferru97/pypaperbot/blob/master/PyPaperBot/Crossref.py"
    click Downloader "https://github.com/ferru97/pypaperbot/blob/master/PyPaperBot/Downloader.py"
    click HTMLParser "https://github.com/ferru97/pypaperbot/blob/master/PyPaperBot/HTMLparsers.py"
    click Paper "https://github.com/ferru97/pypaperbot/blob/master/PyPaperBot/Paper.py"
    click Filter "https://github.com/ferru97/pypaperbot/blob/master/PyPaperBot/PapersFilters.py"
    click Utils "https://github.com/ferru97/pypaperbot/blob/master/PyPaperBot/Utils.py"
    click NetProxy "https://github.com/ferru97/pypaperbot/blob/master/PyPaperBot/NetInfo.py"
    click NetProxy "https://github.com/ferru97/pypaperbot/blob/master/PyPaperBot/proxy.py"

    %% Styles
    classDef cli fill:#f9a825,stroke:#f57f17,color:#ffffff;
    classDef main fill:#42a5f5,stroke:#01579b,color:#ffffff;
    classDef module fill:#ffcc80,stroke:#ef6c00,color:#000000;
    classDef external fill:#a5d6a7,stroke:#1b5e20,color:#000000;
    classDef utility fill:#ce93d8,stroke:#6a1b9a,color:#ffffff;
```
