How ali-fartoot/professorconnected works
And Label Main Components
• API Layer & Application Entry Point: Represent the FastAPI server (launched via app.py) that exposes endpoints for adding professors, cleaning the database, and executing search operations. Indicate that this layer handles incoming HTTP requests (POST for add/search, DELETE for cleanup) and orchestrates other components.
• Crawler Module: Clearly display the crawler component (found in the “crawler” directory). This module interfaces with arXiv to fetch professor-related publications.
• Paper Processing Pipeline: Break down this pipeline into the specific stages as described in the README:
• Title Analysis
• Text Summarization (processing sections like introduction and conclusion)
• Keyword Extraction (with three sub-methods: N-gram based, BERT-based, and LLM-based extraction followed by a revision stage)
• Embedding Stage & Vector Generation: Include a component that captures the embedding process wherein the processed text (concatenated from titles, summaries, and keywords) is converted into vector representations.
• Vector Database & Hybrid Search: Represent the vector search module (the “vector_search” directory) which interacts with a vector database to store embeddings and perform cosine similarity-based, hybrid search and re-ranking. This also connects back to the professor database component (indicated in the “professor_db” directory).
Relationships and Data Flow
• The API triggers the Crawler which fetches raw data from arXiv.
• Fetched data flows into the Paper Processing Pipeline where it is analyzed (Title, Summarization, Keyword Extraction).
• Processed outputs are then concatenated and sent for Embedding.
• The resulting embedding vectors are stored in, or queried from, the Vector Database.
• Finally, the hybrid search module retrieves and re-ranks similar professors based on cosine similarity and keyword overlap.
• Control flows (e.g., re-ranking operations based on a computed keyword score).
Architectural Patterns & Technologies
• Point out that the system follows a microservices/API-based pattern where different modules (crawler, processor, vector search) have dedicated responsibilities.
• Usage of external APIs (arXiv for publications) and external services (e.g., LLM inference via Mistral-Nemo 14B which can run in GPU-accelerated mode).
• Use of shell scripts (in the “scripts” directory) for managing operations like adding, deleting, and searching in the database. (These scripts underscore the operational aspects of your API-driven service.)
Essential Details Without Overcomplication
• Be detailed: add as many modules as possible (each separate script, database file, and test indicator can be abstracted as supporting components if needed).
• Do not shy away from showing the control flow—especially where user input might trigger “cold start” routines (using cold_start.py) or API-based interactions (e.g., add, delete, search).