# anduriroshan/medscraper architecture

> Architecture diagram of the GitHub repository anduriroshan/medscraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anduriroshan/medscraper
- Repository: https://github.com/anduriroshan/medscraper

## Overview

##### Main Components
- Web Crawler: Represented by the module (crawler.py) responsible for scraping articles from the Nature website.
- Article Storage (MySQL): Show the MySQL database where articles (titles, abstracts, dates, summaries, and keywords) are stored. Include a connection link from the crawler and summarization modules.
- Summarization & Keyword Extraction Module: Represented by the summarization.py (and integrated TF-IDF for keyword extraction) that processes abstracts with the Hugging Face summarization pipeline.
- Embedding Generation & Vector Search: Show the vector.py module generating embeddings using SentenceTransformer, and the Milvus vector database that stores these embeddings to support semantic search.
- Natural Language Query Module: Represented by query_search.py, which utilizes Spacy for processing date-related expressions in queries and then uses the SentenceTransformer’s embedding functionality coupled with Milvus for vector similarity search.
- Orchestration/Execution: The run_all.sh or setup.py might be used to tie together all modules and trigger the complete sequence.

##### Relationships and Interactions
- Data Flow from Crawler → MySQL: Crawled article data (title, publication date, abstract) flows from the crawler module into MySQL.
- Processing Flow from MySQL → Summarization/Keyword Extraction: Abstracts stored in MySQL are then processed by the summarization module. The generated summaries and keywords are updated back into MySQL.
- Embedding Flow from MySQL → Vector Generation → Milvus: Embeddings generated from article titles via SentenceTransformer are inserted into Milvus (as per the vector.py module). This creates a searchable index.
- User Query Flow: Natural language queries are handled by query_search.py – following processing with Spacy (extracting time/date entities) and embedding generation (via SentenceTransformer) to perform a vector similarity search in Milvus, then filtering articles in MySQL based on date constraints.
- Logging and Configuration: The logger.py and config module indicate that logging and configuration settings (from config.ini) manage runtime settings, error reporting, and environment-specific parameters.

##### Architectural Patterns and Design Principles
- Modular Design: Each Python module (crawler.py, summarization.py, vector.py, query_search.py) represents a logical unit; the system works in a modular fashion where individual components (parsing, processing, and storage) interact over well-defined interfaces.
- Data Flow Orientation: Emphasize the sequential/process flow of data – first crawl, then process/store, then query.

##### Technologies and Frameworks to Highlight
- Python as the core language.
- MySQL for relational article storage.
- Milvus as the vector database.
- Hugging Face Transformers for text summarization.
- SentenceTransformer for embedding generation.
- Spacy for NLP query parsing.
- ThreadPoolExecutor for parallelizing summarization tasks (where applicable).

##### Additional Elements
- Startup or orchestration script (run_all.sh) as an initiator for the workflow.
- Optionally, include external technologies (e.g., external APIs like Nature’s website, if needed for clarity).

## Diagram (Mermaid)

```mermaid
flowchart TD
    NATURE["Nature Website"]:::external
    ORCHESTRATION["Orchestration"]:::orchestration
    CRAWLER["Article Crawler"]:::crawler
    MYSQL["MySQL Database"]:::database
    SUMMARIZATION["Summarization & Keyword Extraction"]:::processing
    EMBEDDING["Embedding Generation"]:::vector
    MILVUS["Milvus Vector DB"]:::vector
    QUERY["Natural Language Query Processor"]:::query

    subgraph "Support Services"
        LOGGER["Logger"]:::config
        CONFIG["Configuration"]:::config
    end

    %% Data Flow
    NATURE -->|"scrapes"| CRAWLER
    CRAWLER -->|"stores_data"| MYSQL
    MYSQL -->|"sends_abstracts"| SUMMARIZATION
    SUMMARIZATION -->|"updates_summaries"| MYSQL
    MYSQL -->|"sends_titles"| EMBEDDING
    EMBEDDING -->|"stores_embeddings"| MILVUS
    QUERY -->|"vector_search"| MILVUS
    QUERY -->|"filters_by_date"| MYSQL

    %% Orchestration triggers
    ORCHESTRATION -->|"triggers"| CRAWLER
    ORCHESTRATION -->|"triggers"| SUMMARIZATION
    ORCHESTRATION -->|"triggers"| EMBEDDING
    ORCHESTRATION -->|"triggers"| QUERY

    %% Support Services integration (dashed lines)
    CRAWLER -.-> LOGGER
    CRAWLER -.-> CONFIG
    SUMMARIZATION -.-> LOGGER
    SUMMARIZATION -.-> CONFIG
    EMBEDDING -.-> LOGGER
    EMBEDDING -.-> CONFIG
    QUERY -.-> LOGGER
    QUERY -.-> CONFIG
    ORCHESTRATION -.-> LOGGER
    ORCHESTRATION -.-> CONFIG

    %% Click Events
    click CRAWLER "https://github.com/anduriroshan/medscraper/blob/main/src/crawler.py"
    click MYSQL "https://github.com/anduriroshan/medscraper/blob/main/data.sql"
    click SUMMARIZATION "https://github.com/anduriroshan/medscraper/blob/main/src/summarization.py"
    click EMBEDDING "https://github.com/anduriroshan/medscraper/blob/main/src/vector.py"
    click QUERY "https://github.com/anduriroshan/medscraper/blob/main/src/query_search.py"
    click ORCHESTRATION "https://github.com/anduriroshan/medscraper/blob/main/src/run_all.sh"
    click LOGGER "https://github.com/anduriroshan/medscraper/blob/main/src/logger.py"
    click CONFIG "https://github.com/anduriroshan/medscraper/blob/main/src/config/config.ini"

    %% Styles
    classDef crawler fill:#F9E79F,stroke:#5D6D7E,stroke-width:2px;
    classDef database fill:#A9DFBF,stroke:#196F3D,stroke-width:2px;
    classDef processing fill:#AED6F1,stroke:#2471A3,stroke-width:2px;
    classDef vector fill:#D7BDE2,stroke:#6C3483,stroke-width:2px;
    classDef query fill:#F5CBA7,stroke:#D35400,stroke-width:2px;
    classDef orchestration fill:#FAD7A0,stroke:#D68910,stroke-width:2px;
    classDef config fill:#E8DAEF,stroke:#8E44AD,stroke-width:2px;
    classDef external fill:#D5F5E3,stroke:#27AE60,stroke-width:2px;
```
