How harvard-lil/warc-gpt works
Main Components
a. Data Layer:
• WARC Files Storage (./warc directory)
• Vector Store (ChromaDB)
b. Processing Pipeline:
• Ingestion System
• Embedding Generation System
• Text Extraction System (for HTML and PDF)
c. Application Layer:
• Web Server (Flask)
• REST API
• Web UI
• Visualization System
d. External Services:
• LLM Providers (OpenAI API and/or Ollama)
• Embedding Models
Component Relationships and Data Flow
[Data Layer] -> [Processing Pipeline] -> [Application Layer] -> [External Services]
Specific Implementation Details
a. Data Processing Flow:
• WARC Files → Text Extraction → Embedding Generation → Vector Store
• Parallel paths for HTML and PDF processing
b. Runtime Flow:
• Web UI → Flask Server → API Endpoints
• Search Flow: Query → Vector Store → Retrieved Contexts
• Completion Flow: Query + Contexts → LLM → Response
Key Elements to Highlight
a. API Endpoints:
• /api/models
• /api/search
• /api/complete
b. Main Commands:
• ingest
c. UI Components:
• Chat Interface
• Settings Dialog
• Inspection Dialog
• Visualization View
Important Architectural Notes
• Modular nature of LLM providers (OpenAI/Ollama)
• Asynchronous nature of the completion streaming
• RAG pipeline flow
• Separation between ingestion and runtime processes
Technical Details
• Python 3.11+ requirement
• Flask web framework
• ChromaDB for vector storage
• Environment-based configuration
• WebSocket or SSE for streaming responses
External Services Layer (LLMs)
This architecture should emphasize the system's flexibility in terms of LLM providers and embedding models while maintaining a clear separation of concerns between data processing, storage, and retrieval components.