How anduriroshan/semantic-article-finder works
Main Components
• Article Crawler: Represented by the src/crawler.py module that scrapes articles from Nature’s Oncology webpage.
• Data Storage:
• MySQL Database for storing crawled articles, abstracts, summaries, keywords, and metadata.
• The “data.sql” file and nature_articles.db indicate sample or historical data.
• Text Processing & Summarization:
• The src/summarization.py module uses Hugging Face Transformers (e.g., T5 or DistilBART) to generate summaries.
• TF-IDF is used for keyword extraction.
• Vector Embedding & Search:
• The src/vector.py module employs Sentence Transformers to create embeddings for article titles.
• Milvus, the vector database, is used to store and search these embeddings.
• Query Processing:
• The src/query_search.py module that interprets user queries (using Spacy for extracting time-related expressions) and performs search by combining MySQL filtering with Milvus vector search.
• Logger/Configuration:
• src/logger.py for logging and debug messages.
• src/config/config.ini for configuration settings.
• Parallel Processing:
• Use of ThreadPoolExecutor for processing tasks (such as summarization) concurrently.
Relationships and Data Flow
• It scrapes articles, then writes the article data (titles, abstracts, publication dates) into the MySQL database.
• Text Summarization Module:
• Reads articles from MySQL, processes abstracts to generate summaries and extracts keywords.
• Updates MySQL with summaries and metadata.
• Vector Module:
• Creates embeddings for article titles using Sentence Transformers.
• Inserts these embeddings into the Milvus vector database.
• Query Processing Module:
• When a user sends a query, it is parsed (with Spacy) to extract date/time patterns.
• Query is also transformed into an embedding.
• The system then performs a vector search in Milvus, and further refines results by filtering through MySQL based on publication dates.
• All components rely on configuration settings from the config file in src/config.
Architectural Patterns and Design Principles
• Separation of Concerns:
• Each module is clearly responsible for a distinct task: crawling, summarization, vector indexing, and query search.
• Modular Design:
• The code in the src directory is split into self-contained modules to ease maintenance.
• Asynchronous/Parallel Processing:
• The use of ThreadPoolExecutor to speed up tasks like summarization shows a focus on performance for handling larger datasets.
• Integration of Multiple Technologies:
• Clear interfacing between relational (MySQL) and non-relational (Milvus) databases, along with integration of third-party APIs (Hugging Face, Spacy).