How sayan112207/perplexa works
Overall Project Type and Purpose
• The project is a smart web-searching AI that implements a Retrieval Augmented Generation (RAG) architecture. Its purpose is to handle user queries by scraping search results (using DuckDuckGo), validating and summarizing content, and then generating well-structured responses with the help of a local AI model (Qwen2) hosted via Ollama.
• The project is “full-stack” in nature since it combines backend processing for web scraping and AI responses with user-facing interfaces (evidenced by the Gradio implementation in the Perplexa Research Mode notebooks).
Analyzing the File Structure
• Key files and directories include:
• README.md: Contains the high-level architectural overview and detailed process description.
• app.py: Likely a main application file that possibly sets up the core web or API server.
• bot.py: Implements the bot’s conversational/query processing logic.
• Perplexa_Research_Mode.ipynb and Perplexity.ipynb: Notebooks providing interactive research and demonstration environments.
• Configuration directories (.devcontainer, .github) indicate development and deployment workflows.
• The file tree shows an emphasis on a streamlined codebase where most core functionalities (such as query processing, web scraping, content summarization, and AI generation) are embedded in relatively few files.
a. Main Components:
• User Interface: This can include both the bot interaction (via bot.py) and the web interface (possibly built in app.py and the Gradio notebooks).
• Query Processing Module: Responsible for analyzing user questions and extracting keywords.
• Web Scraping Module: Utilizes libraries like Requests and BeautifulSoup to fetch search results from DuckDuckGo.
• Content Validation and Summarization Module: Filters and condenses scraped content.
• AI Generation Module: Engages with the Qwen2 language model via Ollama to produce AI-powered responses.
• Database: MongoDB, which stores persistent data as noted in the README.
• External Services:
• Ollama service (for AI model inference).
• Web search endpoints (DuckDuckGo scraping).
b. Interactions and Data Flow:
• The user input flowing into the Query Processing Module.
• Data from the Query Processing Module being used to trigger the Web Scraping Module.
• Scraped data moving into the Content Validation/Summarization Module.
• The validated and summarized data then being passed, along with the query context, to the AI Generation Module.
• The final response being routed back to the user interface.
c. Architectural Patterns:
• The system splits concerns clearly, following a pipeline style pattern, where each step (Retrieval, Augmentation, Generation) is a distinct module.
• There is an inherent separation of concerns between asynchronous operations (e.g., web scraping and multi-threading used for concurrent operations) and sequential query-response handling.
d. Technologies and Libraries:
• Python (as the programming language)
• Requests & BeautifulSoup for web scraping.
• Threading for concurrent processing.
• Ollama and the Qwen2 model for AI generation.
• MongoDB as the backend database.
• Gradio for UI (specifically in the Research Mode).