# ali-fartoot/professorconnected architecture

> Architecture diagram of the GitHub repository ali-fartoot/professorconnected, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ali-fartoot/professorconnected
- Repository: https://github.com/ali-fartoot/professorconnected

## Overview

##### And Label Main Components
- API Layer & Application Entry Point: Represent the FastAPI server (launched via app.py) that exposes endpoints for adding professors, cleaning the database, and executing search operations. Indicate that this layer handles incoming HTTP requests (POST for add/search, DELETE for cleanup) and orchestrates other components.
- Crawler Module: Clearly display the crawler component (found in the “crawler” directory). This module interfaces with arXiv to fetch professor-related publications.
- Paper Processing Pipeline: Break down this pipeline into the specific stages as described in the README:
- Title Analysis
- Text Summarization (processing sections like introduction and conclusion)
- Keyword Extraction (with three sub-methods: N-gram based, BERT-based, and LLM-based extraction followed by a revision stage)
- Embedding Stage & Vector Generation: Include a component that captures the embedding process wherein the processed text (concatenated from titles, summaries, and keywords) is converted into vector representations.
- Vector Database & Hybrid Search: Represent the vector search module (the “vector_search” directory) which interacts with a vector database to store embeddings and perform cosine similarity-based, hybrid search and re-ranking. This also connects back to the professor database component (indicated in the “professor_db” directory).

##### Relationships and Data Flow
- The API triggers the Crawler which fetches raw data from arXiv.
- Fetched data flows into the Paper Processing Pipeline where it is analyzed (Title, Summarization, Keyword Extraction).
- Processed outputs are then concatenated and sent for Embedding.
- The resulting embedding vectors are stored in, or queried from, the Vector Database.
- Finally, the hybrid search module retrieves and re-ranks similar professors based on cosine similarity and keyword overlap.
- Control flows (e.g., re-ranking operations based on a computed keyword score).

##### Architectural Patterns & Technologies
- Point out that the system follows a microservices/API-based pattern where different modules (crawler, processor, vector search) have dedicated responsibilities.
- Usage of external APIs (arXiv for publications) and external services (e.g., LLM inference via Mistral-Nemo 14B which can run in GPU-accelerated mode).
- Use of shell scripts (in the “scripts” directory) for managing operations like adding, deleting, and searching in the database. (These scripts underscore the operational aspects of your API-driven service.)

##### Essential Details Without Overcomplication
- Be detailed: add as many modules as possible (each separate script, database file, and test indicator can be abstracted as supporting components if needed).
- Do not shy away from showing the control flow—especially where user input might trigger “cold start” routines (using cold_start.py) or API-based interactions (e.g., add, delete, search).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Nodes
    Client["Client Request"]:::external

    API["FastAPI Server (app.py)"]:::api

    %% Crawler Module Subgraph
    subgraph "Crawler Module"
        CrawlerCore["Crawler Core"]:::crawler
        CrawlerInit["__init__.py (crawler)"]:::crawler
        CrawlerUtils["utils.py (crawler)"]:::crawler
    end

    %% External arXiv API
    Arxiv["arXiv API"]:::external

    %% Paper Processing Pipeline Subgraph
    subgraph "Paper Processing Pipeline"
        Title["Title Analysis"]:::processing
        Summarization["Text Summarization"]:::processing
        Keyword["Keyword Extraction"]:::processing
        LLMExtractor["LLM Extractor (pdf_to_text/llm_extractor.py)"]:::processing
        Embedding["Embedding Stage"]:::processing
    end

    %% External LLM Inference Server
    ExternalLLM["LLM Inference Server"]:::external

    %% Vector Search Module Subgraph
    subgraph "Vector Search Module"
        VSCore["Vector Search Core"]:::vector
        VSInit["__init__.py (vector_search)"]:::vector
        VSVis["visulizer.py (vector_search)"]:::vector
    end

    %% Professor Database
    ProfessorDB["Professor Database (professor_db)"]:::db

    %% Operational Scripts Subgraph
    subgraph "Operational Scripts"
        AddScript["Add Script (scripts/add.sh)"]:::scripts
        DeleteScript["Delete Script (scripts/delete.sh)"]:::scripts
        SearchScript["Search Script (scripts/search.sh)"]:::scripts
    end

    %% Cold Start Operation
    ColdStart["Cold Start Operation (cold_start.py)"]:::scripts

    %% Data Flow
    Client -->|"sends request"| API
    API -->|"triggers"| CrawlerCore
    API -->|"initiates"| Title
    CrawlerCore -->|"fetches from"| Arxiv
    Title -->|"next"| Summarization
    Summarization -->|"next"| Keyword
    Keyword -->|"next"| LLMExtractor
    LLMExtractor -->|"calls"| ExternalLLM
    ExternalLLM -->|"returns"| LLMExtractor
    LLMExtractor -->|"processes into"| Embedding
    Embedding -->|"indexes into"| VSCore
    VSCore -->|"queries"| ProfessorDB

    %% Operational & Cold Start Flow (Administrative)
    API ---|"controls"| AddScript
    API ---|"controls"| DeleteScript
    API ---|"controls"| SearchScript
    API ---|"triggers"| ColdStart

    %% Internal Connections within Crawler Module
    CrawlerCore ---|"includes"| CrawlerInit
    CrawlerCore ---|"includes"| CrawlerUtils

    %% Internal Connections within Vector Search Module
    VSCore ---|"includes"| VSInit
    VSCore ---|"includes"| VSVis

    %% Click Events
    click API "https://github.com/ali-fartoot/professorconnected/blob/main/app.py"
    click CrawlerCore "https://github.com/ali-fartoot/professorconnected/tree/main/crawler/"
    click CrawlerInit "https://github.com/ali-fartoot/professorconnected/blob/main/crawler/__init__.py"
    click CrawlerUtils "https://github.com/ali-fartoot/professorconnected/blob/main/crawler/utils.py"
    click LLMExtractor "https://github.com/ali-fartoot/professorconnected/blob/main/pdf_to_text/llm_extractor.py"
    click VSCore "https://github.com/ali-fartoot/professorconnected/tree/main/vector_search/"
    click VSInit "https://github.com/ali-fartoot/professorconnected/blob/main/vector_search/__init__.py"
    click VSVis "https://github.com/ali-fartoot/professorconnected/blob/main/vector_search/visulizer.py"
    click ProfessorDB "https://github.com/ali-fartoot/professorconnected/tree/main/professor_db/"
    click AddScript "https://github.com/ali-fartoot/professorconnected/blob/main/scripts/add.sh"
    click DeleteScript "https://github.com/ali-fartoot/professorconnected/blob/main/scripts/delete.sh"
    click SearchScript "https://github.com/ali-fartoot/professorconnected/blob/main/scripts/search.sh"
    click ColdStart "https://github.com/ali-fartoot/professorconnected/blob/main/cold_start.py"

    %% Styles
    classDef api fill:#DDEBF7,stroke:#2F75B5,stroke-width:2px;
    classDef crawler fill:#FCE4D6,stroke:#B45F06,stroke-width:2px;
    classDef processing fill:#E2EFDA,stroke:#548235,stroke-width:2px;
    classDef vector fill:#FFF2CC,stroke:#BF9000,stroke-width:2px;
    classDef db fill:#F8CBAD,stroke:#C65911,stroke-width:2px;
    classDef scripts fill:#D9E1F2,stroke:#1155CC,stroke-width:2px;
    classDef external fill:#EEE,stroke:#999,stroke-width:2px;
```
