How anduriroshan/medscraper works
Main Components
• Web Crawler: Represented by the module (crawler.py) responsible for scraping articles from the Nature website.
• Article Storage (MySQL): Show the MySQL database where articles (titles, abstracts, dates, summaries, and keywords) are stored. Include a connection link from the crawler and summarization modules.
• Summarization & Keyword Extraction Module: Represented by the summarization.py (and integrated TF-IDF for keyword extraction) that processes abstracts with the Hugging Face summarization pipeline.
• Embedding Generation & Vector Search: Show the vector.py module generating embeddings using SentenceTransformer, and the Milvus vector database that stores these embeddings to support semantic search.
• Natural Language Query Module: Represented by query_search.py, which utilizes Spacy for processing date-related expressions in queries and then uses the SentenceTransformer’s embedding functionality coupled with Milvus for vector similarity search.
• Orchestration/Execution: The run_all.sh or setup.py might be used to tie together all modules and trigger the complete sequence.
Relationships and Interactions
• Data Flow from Crawler → MySQL: Crawled article data (title, publication date, abstract) flows from the crawler module into MySQL.
• Processing Flow from MySQL → Summarization/Keyword Extraction: Abstracts stored in MySQL are then processed by the summarization module. The generated summaries and keywords are updated back into MySQL.
• Embedding Flow from MySQL → Vector Generation → Milvus: Embeddings generated from article titles via SentenceTransformer are inserted into Milvus (as per the vector.py module). This creates a searchable index.
• User Query Flow: Natural language queries are handled by query_search.py – following processing with Spacy (extracting time/date entities) and embedding generation (via SentenceTransformer) to perform a vector similarity search in Milvus, then filtering articles in MySQL based on date constraints.
• Logging and Configuration: The logger.py and config module indicate that logging and configuration settings (from config.ini) manage runtime settings, error reporting, and environment-specific parameters.
Architectural Patterns and Design Principles
• Modular Design: Each Python module (crawler.py, summarization.py, vector.py, query_search.py) represents a logical unit; the system works in a modular fashion where individual components (parsing, processing, and storage) interact over well-defined interfaces.
• Data Flow Orientation: Emphasize the sequential/process flow of data – first crawl, then process/store, then query.
Technologies and Frameworks to Highlight
• Python as the core language.
• MySQL for relational article storage.
• Milvus as the vector database.
• Hugging Face Transformers for text summarization.
• SentenceTransformer for embedding generation.
• Spacy for NLP query parsing.
• ThreadPoolExecutor for parallelizing summarization tasks (where applicable).
Additional Elements
• Startup or orchestration script (run_all.sh) as an initiator for the workflow.
• Optionally, include external technologies (e.g., external APIs like Nature’s website, if needed for clarity).