# jina-ai/reader architecture

> Architecture diagram of the GitHub repository jina-ai/reader, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jina-ai/reader
- Repository: https://github.com/jina-ai/reader

## Overview

##### Main Components Identification

a. Client-Facing Layer:
- Two main endpoints: r.jina.ai (URL reading) and s.jina.ai (web search)
- API Gateway handling request headers and routing

b. Core Services:
- Adaptive Crawler Service (backend/functions/src/cloud-functions/adaptive-crawler.ts)
- Main Crawler Service (backend/functions/src/cloud-functions/crawler.ts)
- Search Service (backend/functions/src/cloud-functions/searcher.ts)
- Data Processing Service (backend/functions/src/cloud-functions/data-crunching.ts)

c. Supporting Services:
- Alt Text Generation Service (backend/functions/src/services/alt-text.ts)
- Brave Search Integration (backend/functions/src/services/brave-search.ts)
- PDF Extraction Service (backend/functions/src/services/pdf-extract.ts)
- Web Rendering Service (backend/functions/src/services/puppeteer.ts)

d. Data Storage:
- Firebase/Firestore (indicated by .firebaserc and firestore.rules)
- Caching Layer (mentioned in README for response caching)

**[Client Layer]**

**[API Gateway/Load Balancer]**

**[Request Processing Layer]**
- Header Processing
- URL Validation
- Request Routing

**[Core Services Layer]**
- Adaptive Crawler
- Main Crawler
- Search Service
- Data Processing

**[Supporting Services Layer]**
- Alt Text Generation
- Brave Search
- PDF Extraction
- Puppeteer Rendering

**[Storage Layer]**
- Firebase/Firestore
- Cache

##### Technical Details

Annotations for:
- Node.js v18 runtime
- Firebase deployment
- Puppeteer for SPA handling
- Timeout and selector configurations
- Rate limiting information

##### Integration Points

Connections with:
- External search APIs (Brave Search)
- Vision Language Models (for image captioning)
- Firebase services
- Content delivery networks

This architecture should emphasize the system's ability to handle both simple URL processing and complex operations like SPA rendering, PDF extraction, and image captioning, while maintaining high availability and performance through proper caching and distributed processing.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client Layer
    subgraph Clients
        R["r.jina.ai"]:::client
        S["s.jina.ai"]:::client
    end

    %% API Gateway Layer
    AG["API Gateway/Load Balancer"]:::gateway

    %% Request Processing Layer
    subgraph RequestProcessing
        HP["Header Processing"]:::process
        UV["URL Validation"]:::process
        RR["Request Routing"]:::process
    end

    %% Core Services Layer
    subgraph CoreServices
        AC["Adaptive Crawler"]:::core
        MC["Main Crawler"]:::core
        SS["Search Service"]:::core
        DP["Data Processing"]:::core
    end

    %% Supporting Services Layer
    subgraph SupportServices
        ATG["Alt Text Generation"]:::support
        BS["Brave Search"]:::support
        PE["PDF Extraction"]:::support
        WR["Web Rendering"]:::support
        RF["Response Formatter"]:::support
    end

    %% Storage Layer
    subgraph Storage
        FB["Firebase/Firestore"]:::storage
        Cache["Cache Layer"]:::cache
    end

    %% Database Models
    subgraph Models
        ACT["Adaptive Crawl Task"]:::model
        CR["Crawled"]:::model
        IA["Image Alt"]:::model
        PDF["PDF"]:::model
        SE["Searched"]:::model
    end

    %% Connections
    R & S --> AG
    AG --> HP & UV & RR
    HP & UV & RR --> AC & MC & SS
    AC & MC --> DP
    DP --> ATG & PE & WR
    SS --> BS
    ATG & PE & WR & BS --> RF
    RF --> Cache
    AC & MC & SS --> FB
    FB --> Models

    %% Click Events
    click AC "https://github.com/jina-ai/reader/blob/main/backend/functions/src/cloud-functions/adaptive-crawler.ts"
    click MC "https://github.com/jina-ai/reader/blob/main/backend/functions/src/cloud-functions/crawler.ts"
    click SS "https://github.com/jina-ai/reader/blob/main/backend/functions/src/cloud-functions/searcher.ts"
    click DP "https://github.com/jina-ai/reader/blob/main/backend/functions/src/cloud-functions/data-crunching.ts"
    click ATG "https://github.com/jina-ai/reader/blob/main/backend/functions/src/services/alt-text.ts"
    click BS "https://github.com/jina-ai/reader/blob/main/backend/functions/src/services/brave-search.ts"
    click PE "https://github.com/jina-ai/reader/blob/main/backend/functions/src/services/pdf-extract.ts"
    click WR "https://github.com/jina-ai/reader/blob/main/backend/functions/src/services/puppeteer.ts"
    click FB "https://github.com/jina-ai/reader/blob/main/backend/.firebaserc"
    click ACT "https://github.com/jina-ai/reader/blob/main/backend/functions/src/db/adaptive-crawl-task.ts"
    click CR "https://github.com/jina-ai/reader/blob/main/backend/functions/src/db/crawled.ts"
    click IA "https://github.com/jina-ai/reader/blob/main/backend/functions/src/db/img-alt.ts"
    click PDF "https://github.com/jina-ai/reader/blob/main/backend/functions/src/db/pdf.ts"
    click SE "https://github.com/jina-ai/reader/blob/main/backend/functions/src/db/searched.ts"
    click RF "https://github.com/jina-ai/reader/blob/main/backend/functions/src/services/snapshot-formatter.ts"

    %% Styles
    classDef client fill:#2374ab
    classDef gateway fill:#48a9a6
    classDef process fill:#4ea5d9
    classDef core fill:#44cf6c
    classDef support fill:#e4a013
    classDef storage fill:#8e44ad
    classDef cache fill:#f1c40f
    classDef model fill:#95a5a6

    %% Legend
    subgraph Legend
        L1["Client Components"]:::client
        L2["Gateway"]:::gateway
        L3["Processing"]:::process
        L4["Core Services"]:::core
        L5["Support Services"]:::support
        L6["Storage"]:::storage
        L7["Cache"]:::cache
        L8["Models"]:::model
    end
```
