How mylonasc/arxiv_llm_assistant works
Main Components
• Query/Input Module: Where the user’s interests query is received (as shown in the notebooks and in the sample code snippet in the README).
• ArXiv Retrieval Engine: This part handles fetching papers from arXiv. It includes retrieval logic, re-ranking (using embeddings), and topic modeling components.
• Topic Modeling & Filtering Module: Implements the TFIDF/NMF-based topic modeling and filtering. This module uses enums (e.g., SummaryFocus, FlatDomainOntology) to encode summarization intent.
• Document Generation Engine: That component takes the filtered results and produces a summary document.
• Scheduler & Update Script: A daily job (run_update.sh) calls the retrieval and summarization process and then updates outputs. Additionally, update_website.py refreshes the website with new content.
• Output & Presentation Layer: Producing output files in HTML (stored in directories like outputs/daily and outputs) and static assets (HTML files, script.js, style.css, and images).
• Development/Experimentation Notebooks: Located in the notebooks folder, these are used for iterative development and testing the retrieval and summarization pipelines.
Relationships and Data Flow
• The user or scheduled job triggers the retrieval process (either via notebook code or through an automated script).
• The query is handled by the Retrieval Engine, which performs the searches on arXiv.
• The retrieved papers go through a re-ranking and topic modeling module to filter for relevance.
• The filtered results are fed into the Document Generation Engine to produce a summary document.
• The generated outputs are saved as HTML files in the outputs directory.
• A separate script (update_website.py) uses these outputs to update the associated website (and the homepage index.html and assets in the ‘src/assets’ directory support presentation).
• The modularity is reinforced by having separate Python scripts (arxivhero.py at the top and in src/) that indicate a clear separation between core processing and deployment/presentation layers.
Architectural Patterns and Design Principles
• Separation of Concerns: Each step (retrieval, filtering, summarization, output generation, and website updating) is handled by a dedicated module.
• Modularity and Extensibility: The use of enums for summarization configurations and a pluggable design for different topic modeling strategies (TFIDF, NMF) allow for future extensions.
• Pipeline/Workflow Pattern: Data flows sequentially from retrieval → processing → document generation → publication.
• Batch Processing and Scheduling: Automated job scheduling (run_update.sh) for daily updates reflects batch processing best practices.
Relevant Technologies and Frameworks
• Python as the main development language, with notebooks highlighting interactive development.
• Likely use of libraries for embeddings and topic modeling (such as scikit-learn for TFIDF/NMF) and language generation.
• HTML/JavaScript/CSS for the front-end output; the assets directory under src contains script.js and style.css.
• Shell scripting for automation (run_update.sh).