How sshh12/llm_osint works
Main Components
a) Core Components:
• Knowledge Agent (Central Orchestrator)
• Web Agent (Information Gatherer)
• LLM Interface Layer
• Tool Layer
• Caching System
b) External Services:
• OpenAI API (GPT-4 & GPT-3.5)
• Serper API (Google Search)
• ScrapingBee API (Web Scraping)
┌─ External APIs Layer ─────────────────────────────────┐
• OpenAI API | Serper API | ScrapingBee API │
• ┘
• ↑
┌─ Core Processing Layer ─────────────────────────────┐
• ┌─ Knowledge Agent ─────────────────────────┐ │
• Orchestrates information gathering │ │
• Manages deep dive rounds │ │
• Processes final results │ │
• ┘ │
• ↓ │
• ┌─ Web Agent ────────────────────────────┐ │
• Executes search queries │ │
• Processes webpage content │ │
• Manages tool interactions │ │
• ┘ │
• ┘
• ↑
┌─ Tool Layer ───────────────────────────────────────┐
• ┌─ Search Tool ─┐ ┌─ Read Link Tool ──────────┐ │
• Serper API │ │ - DOM Parsing │ │
• Integration │ │ - Chunk Processing │ │
• ┘ │ - Content Summarization │ │
• ┘ │
• ┘
• ↑
┌─ Cache Layer ──────────────────────────────────────┐
• Stores processed results and webpage content │
• ┘
Data Flow Indicators
Initial Query → Knowledge Agent
• ↓
Initial Web Agent Search
• ↓
Deep Dive Topics Identification
• ↓
Parallel Deep Dive Web Agents
• ↓
Knowledge Base Aggregation
• ↓
Final Response Generation
Component Details
Knowledge Agent:
• Connection to GPT-4
• Deep dive orchestration logic
• Knowledge base management
Web Agent:
• Tool access patterns
• LLM integration points
• Information gathering workflow
Tool Layer:
• API integration points
• Content processing pipeline
• Caching mechanisms
Additional Elements
• Component types
• Special symbols
Annotations for:
• Token usage considerations
• Rate limiting
• Error handling
• Caching strategies
Remember to include the cost optimization aspects and the ability to reuse knowledge bases for multiple queries as key architectural features.
This system design should be maintained at a level that clearly shows the separation between the orchestration layer (Knowledge Agent) and the execution layer (Web Agents), while still providing enough detail about the tool integration and content processing pipeline.