# sayan112207/perplexa architecture

> Architecture diagram of the GitHub repository sayan112207/perplexa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sayan112207/perplexa
- Repository: https://github.com/sayan112207/perplexa

## Overview

##### Overall Project Type and Purpose
- The project is a smart web-searching AI that implements a Retrieval Augmented Generation (RAG) architecture. Its purpose is to handle user queries by scraping search results (using DuckDuckGo), validating and summarizing content, and then generating well-structured responses with the help of a local AI model (Qwen2) hosted via Ollama.
- The project is “full-stack” in nature since it combines backend processing for web scraping and AI responses with user-facing interfaces (evidenced by the Gradio implementation in the Perplexa Research Mode notebooks).

##### Analyzing the File Structure
- Key files and directories include:
- README.md: Contains the high-level architectural overview and detailed process description.
- app.py: Likely a main application file that possibly sets up the core web or API server.
- bot.py: Implements the bot’s conversational/query processing logic.
- Perplexa_Research_Mode.ipynb and Perplexity.ipynb: Notebooks providing interactive research and demonstration environments.
- Configuration directories (.devcontainer, .github) indicate development and deployment workflows.
- The file tree shows an emphasis on a streamlined codebase where most core functionalities (such as query processing, web scraping, content summarization, and AI generation) are embedded in relatively few files.

a. Main Components:
- User Interface: This can include both the bot interaction (via bot.py) and the web interface (possibly built in app.py and the Gradio notebooks).
- Query Processing Module: Responsible for analyzing user questions and extracting keywords.
- Web Scraping Module: Utilizes libraries like Requests and BeautifulSoup to fetch search results from DuckDuckGo.
- Content Validation and Summarization Module: Filters and condenses scraped content.
- AI Generation Module: Engages with the Qwen2 language model via Ollama to produce AI-powered responses.
- Database: MongoDB, which stores persistent data as noted in the README.
- External Services:
- Ollama service (for AI model inference).
- Web search endpoints (DuckDuckGo scraping).
b. Interactions and Data Flow:
- The user input flowing into the Query Processing Module.
- Data from the Query Processing Module being used to trigger the Web Scraping Module.
- Scraped data moving into the Content Validation/Summarization Module.
- The validated and summarized data then being passed, along with the query context, to the AI Generation Module.
- The final response being routed back to the user interface.
c. Architectural Patterns:
- The system splits concerns clearly, following a pipeline style pattern, where each step (Retrieval, Augmentation, Generation) is a distinct module.
- There is an inherent separation of concerns between asynchronous operations (e.g., web scraping and multi-threading used for concurrent operations) and sequential query-response handling.
d. Technologies and Libraries:
- Python (as the programming language)
- Requests & BeautifulSoup for web scraping.
- Threading for concurrent processing.
- Ollama and the Qwen2 model for AI generation.
- MongoDB as the backend database.
- Gradio for UI (specifically in the Research Mode).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Main Pipeline"
        UI1["Research Mode Notebook"]:::frontend
        UI2["Web UI Notebook"]:::frontend
        QP["Query Processing Module"]:::processing
        WS["Web Scraping Module"]:::processing
        CV["Content Validation & Summarization Module"]:::processing
        AI["AI Generation Module"]:::processing
        DB["MongoDB"]:::db
        DDG["DuckDuckGo"]:::external
        OLL["Ollama"]:::external
    end

    UI1 --> QP
    UI2 --> QP
    QP --> WS
    WS -->|"uses"| DDG
    WS --> CV
    CV --> AI
    AI -->|"stores"| DB
    AI -->|"calls"| OLL
    AI -->|"updates"| UI1
    AI -->|"updates"| UI2

    subgraph "Bot Workflow"
        BQ["User Query"]:::frontend
        BR["Input Reception (bot.py)"]:::bot
        BP["Query Preprocessing"]:::processing
        BS["Web Scraping Module"]:::processing
        BC["Content Filtering & Summarization"]:::processing
        BA["Ollama (AI Generation)"]:::processing
        BO["Response Output"]:::frontend
    end

    BQ --> BR
    BR --> BP
    BP --> BS
    BS -->|"BeautifulSoup/Requests"| BC
    BC --> BA
    BA --> BO

    subgraph "Project Configurations"
        DOC["README.md"]:::config
        DC1[".devcontainer"]:::config
        DC2[".github"]:::config
    end

    DOC --> DC1
    DOC --> DC2

    %% Click Events
    click UI1 "https://github.com/sayan112207/perplexa/blob/main/Perplexa_Research_Mode.ipynb"
    click UI2 "https://github.com/sayan112207/perplexa/blob/main/Perplexity.ipynb"
    click QP "https://github.com/sayan112207/perplexa/blob/main/app.py"
    click BR "https://github.com/sayan112207/perplexa/blob/main/bot.py"
    click DOC "https://github.com/sayan112207/perplexa/blob/main/README.md"
    click DC1 "https://github.com/sayan112207/perplexa/blob/main/.devcontainer"
    click DC2 "https://github.com/sayan112207/perplexa/blob/main/.github"

    %% Styles
    classDef frontend fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef processing fill:#98FB98,stroke:#333,stroke-width:2px;
    classDef external fill:#87CEFA,stroke:#333,stroke-width:2px;
    classDef db fill:#FFB6C1,stroke:#333,stroke-width:2px;
    classDef config fill:#D8BFD8,stroke:#333,stroke-width:2px;
    classDef bot fill:#FFA07A,stroke:#333,stroke-width:2px;
```
