# anduriroshan/semantic-article-finder architecture

> Architecture diagram of the GitHub repository anduriroshan/semantic-article-finder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anduriroshan/semantic-article-finder
- Repository: https://github.com/anduriroshan/semantic-article-finder

## Overview

##### Main Components
- Article Crawler: Represented by the src/crawler.py module that scrapes articles from Nature’s Oncology webpage.
- Data Storage:
- MySQL Database for storing crawled articles, abstracts, summaries, keywords, and metadata.
- The “data.sql” file and nature_articles.db indicate sample or historical data.
- Text Processing & Summarization:
- The src/summarization.py module uses Hugging Face Transformers (e.g., T5 or DistilBART) to generate summaries.
- TF-IDF is used for keyword extraction.
- Vector Embedding & Search:
- The src/vector.py module employs Sentence Transformers to create embeddings for article titles.
- Milvus, the vector database, is used to store and search these embeddings.
- Query Processing:
- The src/query_search.py module that interprets user queries (using Spacy for extracting time-related expressions) and performs search by combining MySQL filtering with Milvus vector search.
- Logger/Configuration:
- src/logger.py for logging and debug messages.
- src/config/config.ini for configuration settings.
- Parallel Processing:
- Use of ThreadPoolExecutor for processing tasks (such as summarization) concurrently.

##### Relationships and Data Flow
- It scrapes articles, then writes the article data (titles, abstracts, publication dates) into the MySQL database.
- Text Summarization Module:
- Reads articles from MySQL, processes abstracts to generate summaries and extracts keywords.
- Updates MySQL with summaries and metadata.
- Vector Module:
- Creates embeddings for article titles using Sentence Transformers.
- Inserts these embeddings into the Milvus vector database.
- Query Processing Module:
- When a user sends a query, it is parsed (with Spacy) to extract date/time patterns.
- Query is also transformed into an embedding.
- The system then performs a vector search in Milvus, and further refines results by filtering through MySQL based on publication dates.
- All components rely on configuration settings from the config file in src/config.

##### Architectural Patterns and Design Principles
- Separation of Concerns:
- Each module is clearly responsible for a distinct task: crawling, summarization, vector indexing, and query search.
- Modular Design:
- The code in the src directory is split into self-contained modules to ease maintenance.
- Asynchronous/Parallel Processing:
- The use of ThreadPoolExecutor to speed up tasks like summarization shows a focus on performance for handling larger datasets.
- Integration of Multiple Technologies:
- Clear interfacing between relational (MySQL) and non-relational (Milvus) databases, along with integration of third-party APIs (Hugging Face, Spacy).

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["User Query Interface"]:::interface

    subgraph "Data Collection and Processing"
        B["Article Crawler (src/crawler.py)"]:::module
        C["MySQL Database"]:::datastore
        D["Summarization Module (src/summarization.py)\n(HuggingFace Transformers & TF-IDF)"]:::module
        E["Vector Embedding Module (src/vector.py)\n(Sentence Transformers)"]:::module
        F["Milvus Vector Database"]:::datastore
        G["Query Processing Module (src/query_search.py)\n(Spacy & Embedding conversion)"]:::module
        J["Search Results"]:::interface
    end

    subgraph "Utilities"
        H["Logger (src/logger.py)"]:::integration
        I["Configuration (src/config/config.ini)"]:::integration
    end

    A -->|"query"| G
    B -->|"scrapes_articles"| C
    C -->|"read_articles"| D
    D -->|"update_summaries"| C
    C -->|"fetch_data"| E
    E -->|"store_embeddings"| F
    G -->|"vector_search"| F
    G -->|"filter_by_date"| C
    G -->|"return_results"| J

    I -->|"config"| B
    I -->|"config"| D
    I -->|"config"| E
    I -->|"config"| G

    H -->|"logging"| B
    H -->|"logging"| D
    H -->|"logging"| E
    H -->|"logging"| G

    click B "https://github.com/anduriroshan/semantic-article-finder/blob/main/src/crawler.py"
    click D "https://github.com/anduriroshan/semantic-article-finder/blob/main/src/summarization.py"
    click E "https://github.com/anduriroshan/semantic-article-finder/blob/main/src/vector.py"
    click G "https://github.com/anduriroshan/semantic-article-finder/blob/main/src/query_search.py"
    click H "https://github.com/anduriroshan/semantic-article-finder/blob/main/src/logger.py"
    click I "https://github.com/anduriroshan/semantic-article-finder/blob/main/src/config/config.ini"
    click C "https://github.com/anduriroshan/semantic-article-finder/blob/main/data.sql"
    click C "https://github.com/anduriroshan/semantic-article-finder/blob/main/nature_articles.db"

    classDef module fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef datastore fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef integration fill:#FFD580,stroke:#000,stroke-width:1px;
    classDef interface fill:#FFA500,stroke:#000,stroke-width:1px;
```
