# harvard-lil/warc-gpt architecture

> Architecture diagram of the GitHub repository harvard-lil/warc-gpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/harvard-lil/warc-gpt
- Repository: https://github.com/harvard-lil/warc-gpt

## Overview

##### Main Components

a. Data Layer:
- WARC Files Storage (./warc directory)
- Vector Store (ChromaDB)

b. Processing Pipeline:
- Ingestion System
- Embedding Generation System
- Text Extraction System (for HTML and PDF)

c. Application Layer:
- Web Server (Flask)
- REST API
- Web UI
- Visualization System

d. External Services:
- LLM Providers (OpenAI API and/or Ollama)
- Embedding Models

##### Component Relationships and Data Flow

**[Data Layer] -> [Processing Pipeline] -> [Application Layer] -> [External Services]**

##### Specific Implementation Details

a. Data Processing Flow:
- WARC Files → Text Extraction → Embedding Generation → Vector Store
- Parallel paths for HTML and PDF processing

b. Runtime Flow:
- Web UI → Flask Server → API Endpoints
- Search Flow: Query → Vector Store → Retrieved Contexts
- Completion Flow: Query + Contexts → LLM → Response

##### Key Elements to Highlight

a. API Endpoints:
- /api/models
- /api/search
- /api/complete

b. Main Commands:
- ingest

c. UI Components:
- Chat Interface
- Settings Dialog
- Inspection Dialog
- Visualization View

##### Important Architectural Notes

- Modular nature of LLM providers (OpenAI/Ollama)
- Asynchronous nature of the completion streaming
- RAG pipeline flow
- Separation between ingestion and runtime processes

##### Technical Details

- Python 3.11+ requirement
- Flask web framework
- ChromaDB for vector storage
- Environment-based configuration
- WebSocket or SSE for streaming responses

##### External Services Layer (LLMs)

This architecture should emphasize the system's flexibility in terms of LLM providers and embedding models while maintaining a clear separation of concerns between data processing, storage, and retrieval components.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Data
        WARC[("WARC Files")]
        VectorDB[("Vector Store<br/>(ChromaDB)")]
    end

    subgraph Processing
        Ingest["Ingestion System"]
        TextExtract["Text Extraction"]
        EmbedGen["Embedding Generation"]
        Viz["Visualization System"]
    end

    subgraph WebApp
        Server["Flask Server"]
        subgraph API
            Models["/api/models"]
            Search["/api/search"]
            Complete["/api/complete"]
        end
        subgraph UI
            Chat["Chat Interface"]
            Settings["Settings Dialog"]
            Inspect["Inspection Dialog"]
            ChatInp["Chat Input"]
        end
    end

    subgraph External
        LLM["LLM Providers<br/>(OpenAI/Ollama)"]
        EmbedModel["Embedding Models"]
    end

    %% Data Flow
    WARC --> Ingest
    Ingest --> TextExtract
    TextExtract --> EmbedGen
    EmbedGen --> VectorDB
    VectorDB --> Search
    Search --> Complete
    Complete --> Chat
    Chat --> ChatInp
    ChatInp --> Complete
    Complete -.-> LLM
    EmbedGen -.-> EmbedModel

    %% Component Styling
    classDef storage fill:#90EE90,stroke:#333,stroke-width:2px
    classDef processing fill:#FFA500,stroke:#333,stroke-width:2px
    classDef api fill:#87CEEB,stroke:#333,stroke-width:2px
    classDef ui fill:#FFD700,stroke:#333,stroke-width:2px
    classDef external fill:#DDA0DD,stroke:#333,stroke-width:2px

    class WARC,VectorDB storage
    class Ingest,TextExtract,EmbedGen,Viz processing
    class Models,Search,Complete,Server api
    class Chat,Settings,Inspect,ChatInp ui
    class LLM,EmbedModel external

    %% Click Events
    click WARC "https://github.com/harvard-lil/warc-gpt/tree/main//warc"
    click Ingest "https://github.com/harvard-lil/warc-gpt/blob/main//warc_gpt/commands/ingest.py"
    click Viz "https://github.com/harvard-lil/warc-gpt/blob/main//warc_gpt/commands/visualize.py"
    click Chat "https://github.com/harvard-lil/warc-gpt/blob/main//warc_gpt/static/components/ChatFlow.js"
    click Settings "https://github.com/harvard-lil/warc-gpt/blob/main//warc_gpt/static/components/SettingsDialog.js"
    click Inspect "https://github.com/harvard-lil/warc-gpt/blob/main//warc_gpt/static/components/InspectDialog.js"
    click ChatInp "https://github.com/harvard-lil/warc-gpt/blob/main//warc_gpt/static/components/ChatInput.js"
    click Models "https://github.com/harvard-lil/warc-gpt/blob/main//warc_gpt/views/api/models.py"
    click Search "https://github.com/harvard-lil/warc-gpt/blob/main//warc_gpt/views/api/search.py"
    click Complete "https://github.com/harvard-lil/warc-gpt/blob/main//warc_gpt/views/api/complete.py"
    click Server "https://github.com/harvard-lil/warc-gpt/blob/main//warc_gpt/views/ui.py"
```
