# spyglass-search/spyglass architecture

> Architecture diagram of the GitHub repository spyglass-search/spyglass, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/spyglass-search/spyglass
- Repository: https://github.com/spyglass-search/spyglass

## Overview

##### Main System Components

a) Frontend Layer:
- Desktop Client (Tauri-based)
- Search Interface
- Settings Management
- Wizard/Onboarding Flow

b) Backend Core:
- Search Engine Core
- Document Processing Pipeline
- Crawler System
- Plugin System
- Database Layer

c) Integration Layer:
- External Services Connectors (Google Drive, GitHub, Reddit, etc.)
- Local File System Integration
- Authentication Server

##### Architectural Patterns to Highlight

- Modular Crate Structure (microservices-like organization)
- Plugin Architecture
- Pipeline Processing Pattern
- Event-driven Communication

##### Detailed Component Breakdown

**[Desktop Client (Tauri)] ←→ [Core Engine]**
- ↑                      ↑

**[UI Components] ←→ [RPC Layer] → [Search Pipeline]**
- ↓
**[External Services] ← [Connectors] → [Document Processor]**
- ↓
- [Database Layer]

##### Key Relationships

- Document flow: From source → crawler → processor → index → search results
- Plugin system integration points
- Authentication flows
- Search query path
- Event propagation system

##### Technical Details

- Database schema references (SQLite)
- API endpoints
- Plugin extension points
- File processing pipeline
- Search index structure

##### Implementation Notes

- Rust-based backend components
- React/TypeScript frontend
- Tauri framework integration
- Local storage and processing emphasis
- Plugin system architecture

##### Data Flow Indicators

- User input flow
- Document processing flow
- Search query/response path
- Plugin communication
- External service interaction

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Frontend
        DC["Desktop Client"]:::frontend
        click DC "https://github.com/spyglass-search/spyglass/tree/main/apps/desktop-client"
        SI["Search Interface"]:::frontend
        click SI "https://github.com/spyglass-search/spyglass/tree/main/apps/desktop-client/src/pages/search"
        SM["Settings Management"]:::frontend
        click SM "https://github.com/spyglass-search/spyglass/tree/main/apps/desktop-client/src/pages/settings"
        WF["Wizard/Onboarding"]:::frontend
        click WF "https://github.com/spyglass-search/spyglass/tree/main/apps/desktop-client/src/pages/wizard"
    end

    subgraph Core
        CE["Core Engine"]:::core
        click CE "https://github.com/spyglass-search/spyglass/tree/main/crates/spyglass"
        SEC["Search Engine Core"]:::core
        click SEC "https://github.com/spyglass-search/spyglass/tree/main/crates/spyglass-searcher"
        DPP["Document Processing Pipeline"]:::core
        click DPP "https://github.com/spyglass-search/spyglass/tree/main/crates/spyglass-processor"
        CS["Crawler System"]:::core
        click CS "https://github.com/spyglass-search/spyglass/tree/main/crates/spyglass/src/crawler"
        PS["Plugin System"]:::core
        click PS "https://github.com/spyglass-search/spyglass/tree/main/crates/spyglass/src/plugin"
    end

    subgraph Integration
        ESC["External Services Connectors"]:::integration
        click ESC "https://github.com/spyglass-search/spyglass/tree/main/crates/spyglass/src/connection"
        LFSI["Local File System Integration"]:::integration
        click LFSI "https://github.com/spyglass-search/spyglass/tree/main/crates/spyglass/src/filesystem"
        AS["Authentication Server"]:::integration
        click AS "https://github.com/spyglass-search/spyglass/blob/main/crates/spyglass/src/connection/auth_server.rs"
    end

    subgraph Data
        DL["Database Layer"]:::data
        click DL "https://github.com/spyglass-search/spyglass/tree/main/crates/entities"
        DBM["Database Migrations"]:::data
        click DBM "https://github.com/spyglass-search/spyglass/tree/main/crates/migrations"
    end

    subgraph Communication
        RPC["RPC Layer"]:::comm
        click RPC "https://github.com/spyglass-search/spyglass/tree/main/crates/spyglass-rpc"
        DP["Document Processing"]:::comm
        click DP "https://github.com/spyglass-search/spyglass/tree/main/crates/spyglass-processor/src/parser"
        LLM["LLM Integration"]:::comm
        click LLM "https://github.com/spyglass-search/spyglass/tree/main/crates/spyglass-llm"
    end

    subgraph Extensions
        PA["Plugin Architecture"]:::ext
        click PA "https://github.com/spyglass-search/spyglass/tree/main/plugins"
    end

    %% Frontend Connections
    DC --> RPC
    SI --> DC
    SM --> DC
    WF --> DC

    %% Core Connections
    RPC --> CE
    CE --> SEC
    CE --> DPP
    CE --> CS
    CE --> PS

    %% Integration Connections
    CS --> ESC
    CS --> LFSI
    ESC --> AS

    %% Data Flow
    DPP --> DP
    DP --> DL
    DL --> DBM

    %% Plugin System
    PS --> PA
    PS --> LLM

    %% Communication Flow
    RPC --> DPP
    RPC --> SEC

    classDef frontend fill:#3498db,stroke:#2980b9,color:white
    classDef core fill:#2ecc71,stroke:#27ae60,color:white
    classDef integration fill:#f1c40f,stroke:#f39c12,color:black
    classDef data fill:#95a5a6,stroke:#7f8c8d,color:white
    classDef comm fill:#e74c3c,stroke:#c0392b,color:white
    classDef ext fill:#9b59b6,stroke:#8e44ad,color:white
```
