How ferru97/pypaperbot works
Project’s Main Purpose and Components
• The project is a CLI-driven automated downloader for scholarly papers.
• Its goal is to execute various queries or DOI-based requests to fetch papers and corresponding metadata.
• It interacts with external sources (e.g., Google Scholar, Crossref, SciHub, Annas Archive) and supports features like filtering (by year, journal, etc.) and proxy usage.
Break Down the Application into Core Modules
• Modules for External Data Access:
• Scholar.py: Handles queries to Google Scholar and processes search results.
• Crossref.py: Manages requests to Crossref to retrieve bibtex or metadata.
• Proxy.py & NetInfo.py: Manage network connections and proxy configurations, aiding in accessing blocked resources.
• Modules for Downloading and Parsing:
• Downloader.py: Central logic for downloading papers and associated content (PDFs, bibtex).
• HTMLparsers.py: Implements parsing of HTML content from external websites (e.g., Scholar pages).
• Modules for Data and Filtering:
• Paper.py: Likely encapsulates paper metadata and file naming conventions.
• PapersFilters.py: Provides filtering capabilities based on input criteria (year, citations, journal names).
• Utils.py: Contains helper utilities used across the application.
External Interactions
• Google Scholar (for search queries and citation-based retrieval).
• Crossref service (for metadata and bibtex).
• SciHub/SciDB/Annas Archive (for obtaining the PDF files).
• External systems to the relevant modules (Scholar.py, Crossref.py, Downloader.py).
A Data Flow/Workflow Layer
• Sequence:
1. User invokes PyPaperBot with CLI arguments.
2. __main__.py parses the input and calls appropriate module routines.
3. The Scholar module is queried if the input is search based, which upon receiving responses, passes data to filtering (via PapersFilters.py) and then to Downloader.py.
4. Similarly for DOI-based queries, Crossref.py and Downloader.py take the lead.
5. Utilities (Utils.py) and network management (NetInfo.py/Proxy.py) support all modules.
Architectural Patterns and Technologies
• Modular design: each functionality is encapsulated in its own module, making the system easier to maintain and extend.
• Use of dependency injection aspects – for example, proxy details passed from CLI to the network module.
• Mention that although not explicitly split into a frontend/backend, the application's clear separation of concerns (parsing, downloading, filtering) is its key architectural strength.