# sshh12/llm_osint architecture

> Architecture diagram of the GitHub repository sshh12/llm_osint, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sshh12/llm_osint
- Repository: https://github.com/sshh12/llm_osint

## Overview

##### Main Components

a) Core Components:
- Knowledge Agent (Central Orchestrator)
- Web Agent (Information Gatherer)
- LLM Interface Layer
- Tool Layer
- Caching System

b) External Services:
- OpenAI API (GPT-4 & GPT-3.5)
- Serper API (Google Search)
- ScrapingBee API (Web Scraping)

**┌─ External APIs Layer ─────────────────────────────────┐**
- OpenAI API  |  Serper API  |  ScrapingBee API       │
- ┘
- ↑
**┌─ Core Processing Layer ─────────────────────────────┐**
- ┌─ Knowledge Agent ─────────────────────────┐       │
- Orchestrates information gathering     │       │
- Manages deep dive rounds               │       │
- Processes final results                │       │
- ┘       │
- ↓                               │
- ┌─ Web Agent ────────────────────────────┐         │
- Executes search queries             │         │
- Processes webpage content           │         │
- Manages tool interactions           │         │
- ┘       │
- ┘
- ↑
**┌─ Tool Layer ───────────────────────────────────────┐**
- ┌─ Search Tool ─┐  ┌─ Read Link Tool ──────────┐   │
- Serper API    │  │ - DOM Parsing             │   │
- Integration   │  │ - Chunk Processing        │   │
- ┘  │ - Content Summarization    │   │
- ┘   │
- ┘
- ↑
**┌─ Cache Layer ──────────────────────────────────────┐**
- Stores processed results and webpage content       │
- ┘

##### Data Flow Indicators

**Initial Query → Knowledge Agent**
- ↓
**Initial Web Agent Search**
- ↓
**Deep Dive Topics Identification**
- ↓
**Parallel Deep Dive Web Agents**
- ↓
**Knowledge Base Aggregation**
- ↓
**Final Response Generation**

##### Component Details

Knowledge Agent:
- Connection to GPT-4
- Deep dive orchestration logic
- Knowledge base management

Web Agent:
- Tool access patterns
- LLM integration points
- Information gathering workflow

Tool Layer:
- API integration points
- Content processing pipeline
- Caching mechanisms

##### Additional Elements

- Component types
- Special symbols

Annotations for:
- Token usage considerations
- Rate limiting
- Error handling
- Caching strategies

Remember to include the cost optimization aspects and the ability to reuse knowledge bases for multiple queries as key architectural features.

This system design should be maintained at a level that clearly shows the separation between the orchestration layer (Knowledge Agent) and the execution layer (Web Agents), while still providing enough detail about the tool integration and content processing pipeline.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph External["External APIs Layer"]
        OpenAI["OpenAI API (GPT-4 & 3.5)"]:::external
        Serper["Serper API"]:::external
        ScrapingBee["ScrapingBee API"]:::external
    end

    subgraph Core["Core Processing Layer"]
        KnowledgeAgent["Knowledge Agent<br/>Orchestration & Deep Dives"]:::core
        WebAgent["Web Agent<br/>Information Gathering"]:::core
        LLMInterface["LLM Interface"]:::core
    end

    subgraph Tools["Tool Layer"]
        SearchTool["Search Tool"]:::tool
        ReadLinkTool["Read Link Tool"]:::tool
        LinkProcessor["Link Processor"]:::tool
        ChunkProcessor["Content Chunk<br/>Processor"]:::tool
    end

    subgraph Cache["Cache Layer"]
        CacheSystem["Cache System"]:::cache
    end

    %% Core Layer Connections
    KnowledgeAgent --> WebAgent
    KnowledgeAgent --> LLMInterface
    WebAgent --> LLMInterface

    %% External Layer Connections
    LLMInterface --> OpenAI
    SearchTool --> Serper
    ReadLinkTool --> ScrapingBee

    %% Tool Layer Connections
    WebAgent --> SearchTool
    WebAgent --> ReadLinkTool
    ReadLinkTool --> LinkProcessor
    LinkProcessor --> ChunkProcessor

    %% Cache Connections
    WebAgent --> CacheSystem
    ReadLinkTool --> CacheSystem
    ChunkProcessor --> CacheSystem

    %% Process Flow
    ProcessFlow{{"Process Flow"}}:::flow
    ProcessFlow --> |1. Initial Query| KnowledgeAgent
    KnowledgeAgent --> |2. Search Request| WebAgent
    WebAgent --> |3. Deep Dive Topics| KnowledgeAgent
    KnowledgeAgent --> |4. Knowledge Base| LLMInterface
    LLMInterface --> |5. Final Response| KnowledgeAgent

    %% Styling
    classDef external fill:#90EE90,stroke:#333,stroke-width:2px
    classDef core fill:#87CEEB,stroke:#333,stroke-width:2px
    classDef tool fill:#FFE4B5,stroke:#333,stroke-width:2px
    classDef cache fill:#D3D3D3,stroke:#333,stroke-width:2px
    classDef flow fill:#FFF,stroke:#333,stroke-width:2px

    %% Click Events
    click KnowledgeAgent "https://github.com/sshh12/llm_osint/blob/main/llm_osint/knowledge_agent.py"
    click WebAgent "https://github.com/sshh12/llm_osint/blob/main/llm_osint/web_agent.py"
    click LLMInterface "https://github.com/sshh12/llm_osint/blob/main/llm_osint/llm.py"
    click SearchTool "https://github.com/sshh12/llm_osint/blob/main/llm_osint/tools/search.py"
    click ReadLinkTool "https://github.com/sshh12/llm_osint/blob/main/llm_osint/tools/read_link.py"
    click CacheSystem "https://github.com/sshh12/llm_osint/blob/main/llm_osint/cache_utils.py"
    click LinkProcessor "https://github.com/sshh12/llm_osint/blob/main/llm_osint/link_scraping.py"
    click ChunkProcessor "https://github.com/sshh12/llm_osint/blob/main/llm_osint/llm_map_reduce.py"

    %% Legend
    subgraph Legend
        L1[External APIs]:::external
        L2[Core Components]:::core
        L3[Tools]:::tool
        L4[Cache Components]:::cache
    end
```
