# mylonasc/arxiv_llm_assistant architecture

> Architecture diagram of the GitHub repository mylonasc/arxiv_llm_assistant, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mylonasc/arxiv_llm_assistant
- Repository: https://github.com/mylonasc/arxiv_llm_assistant

## Overview

##### Main Components
- Query/Input Module: Where the user’s interests query is received (as shown in the notebooks and in the sample code snippet in the README).
- ArXiv Retrieval Engine: This part handles fetching papers from arXiv. It includes retrieval logic, re-ranking (using embeddings), and topic modeling components.
- Topic Modeling & Filtering Module: Implements the TFIDF/NMF-based topic modeling and filtering. This module uses enums (e.g., SummaryFocus, FlatDomainOntology) to encode summarization intent.
- Document Generation Engine: That component takes the filtered results and produces a summary document.
- Scheduler & Update Script: A daily job (run_update.sh) calls the retrieval and summarization process and then updates outputs. Additionally, update_website.py refreshes the website with new content.
- Output & Presentation Layer: Producing output files in HTML (stored in directories like outputs/daily and outputs) and static assets (HTML files, script.js, style.css, and images).
- Development/Experimentation Notebooks: Located in the notebooks folder, these are used for iterative development and testing the retrieval and summarization pipelines.

##### Relationships and Data Flow
- The user or scheduled job triggers the retrieval process (either via notebook code or through an automated script).
- The query is handled by the Retrieval Engine, which performs the searches on arXiv.
- The retrieved papers go through a re-ranking and topic modeling module to filter for relevance.
- The filtered results are fed into the Document Generation Engine to produce a summary document.
- The generated outputs are saved as HTML files in the outputs directory.
- A separate script (update_website.py) uses these outputs to update the associated website (and the homepage index.html and assets in the ‘src/assets’ directory support presentation).
- The modularity is reinforced by having separate Python scripts (arxivhero.py at the top and in src/) that indicate a clear separation between core processing and deployment/presentation layers.

##### Architectural Patterns and Design Principles
- Separation of Concerns: Each step (retrieval, filtering, summarization, output generation, and website updating) is handled by a dedicated module.
- Modularity and Extensibility: The use of enums for summarization configurations and a pluggable design for different topic modeling strategies (TFIDF, NMF) allow for future extensions.
- Pipeline/Workflow Pattern: Data flows sequentially from retrieval → processing → document generation → publication.
- Batch Processing and Scheduling: Automated job scheduling (run_update.sh) for daily updates reflects batch processing best practices.

##### Relevant Technologies and Frameworks
- Python as the main development language, with notebooks highlighting interactive development.
- Likely use of libraries for embeddings and topic modeling (such as scikit-learn for TFIDF/NMF) and language generation.
- HTML/JavaScript/CSS for the front-end output; the assets directory under src contains script.js and style.css.
- Shell scripting for automation (run_update.sh).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Triggers
    A["Interactive Query (notebooks/arxiv_retrieve.ipynb)"]:::orange
    B["Scheduler Trigger (run_update.sh)"]:::green

    %% Processing Pipeline
    C["ArXiv Retrieval Engine (arxivhero.py; src/arxiv_hero.py)"]:::blue
    D["Document Generation Engine"]:::blue
    E["Output Reports (outputs/)"]:::orange
    F["Website Update Module (update_website.py)"]:::orange
    G["Homepage (index.html)"]:::orange
    H["Assets (src/assets)"]:::orange

    %% Development / Experimentation
    I["Notebooks (development)"]:::purple

    %% Data Flow
    A -->|"triggers"| C
    B -->|"triggers"| C
    C -->|"re-ranking&topic-modeling"| D
    D -->|"generates summary"| E
    E -->|"publishes report"| F
    F -->|"updates"| G
    F -->|"updates"| H
    I -->|"experimental tests"| A

    %% Click Events
    click A "https://github.com/mylonasc/arxiv_llm_assistant/blob/master/notebooks/arxiv_retrieve.ipynb"
    click B "https://github.com/mylonasc/arxiv_llm_assistant/blob/master/run_update.sh"
    click C "https://github.com/mylonasc/arxiv_llm_assistant/blob/master/arxivhero.py"
    click C "https://github.com/mylonasc/arxiv_llm_assistant/blob/master/src/arxiv_hero.py"
    click E "https://github.com/mylonasc/arxiv_llm_assistant/tree/master/outputs/"
    click F "https://github.com/mylonasc/arxiv_llm_assistant/blob/master/update_website.py"
    click G "https://github.com/mylonasc/arxiv_llm_assistant/blob/master/index.html"
    click H "https://github.com/mylonasc/arxiv_llm_assistant/tree/master/src/assets"
    click I "https://github.com/mylonasc/arxiv_llm_assistant/tree/master/notebooks/"

    %% Styles
    classDef blue fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef green fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef orange fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef purple fill:#e2d6f3,stroke:#6f42c1,stroke-width:2px;
```
