# mishushakov/llm-scraper architecture

> Architecture diagram of the GitHub repository mishushakov/llm-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mishushakov/llm-scraper
- Repository: https://github.com/mishushakov/llm-scraper

## Overview

##### Project Type and Purpose
- The LLM Scraper is an extraction library that connects a webpage’s content with LLM processing. It is not a full-stack web app but rather a dedicated scraping tool. Its purpose is to abstract the complexity of scraping web pages and converting them into structured, type-safe objects (validated by Zod) using an LLM.
- Key use-cases include scraping, structured data extraction, code generation for scraping scripts, and streaming data extraction.

##### File Structure
- Top-level files include configuration files (.gitignore, .prettierrc, tsconfig.json, package.json), which confirm this is a Node/TypeScript project.
- The “src” directory contains the core library code (e.g., LLMScraper’s main implementation in index.ts, models.ts, and cleanup.ts).
- The “examples” directory provides usage examples (codegen.ts, hn.ts, local.ts, ollama.ts, streaming.ts) that illustrate different functionalities and providers.
- This clean separation between source code and examples suggests a library design intended for developers to integrate into their projects.

##### README for Additional Insights
- The README explains how to install and use the scraper with various LLM providers.
- It outlines the different modes of operation (standard scraping, streaming, and code-generation) and demonstrates how the Playwright framework is used to manage browser interactions.
- The examples include multiple integration patterns: initializing LLMs, scraping with defined schemas (using Zod), and generating scraping code.

##### Main Components
a. Client Application (User Code):
- The consumer’s code that sets up and utilizes the LLM Scraper (as seen in the examples).
- This includes initiating a Playwright browser and passing a page instance to the scraper.

b. LLMScraper Library:
- The core module (found in src/) that handles the overall logic, taking a provided LLM instance to process page content.
- Inside this module, include separate tasks for cleanup, schema validation, and integration with Playwright.

c. LLM Providers:
- Abstraction for various LLM services (OpenAI, Ollama, GGUF, etc.).
- How the user code can pass different providers to the scraper constructor, emphasizing support for multiple platforms.

d. Playwright:
- The browser automation tool used to navigate to webpages. It manages page creation, navigation, and content extraction.

e. Code Generation Component:
- An optional feature within the LLMScraper that generates scraping scripts based on provided schemas.
- This should be shown as an extension of the base functionality.

f. Data Schema Validation:
- Integration with Zod which ensures that the data returned by the LLM meets the expected structured schema defined by the user.

##### Relationships and Interactions
- From the Client Application to Playwright (launching and managing a browser page).
- From the Client Application to the LLMScraper, showing initialization with an LLM provider.
- From the LLMScraper to the LLM Provider, representing the data extraction using function calling under the hood.
- From the LLMScraper to Zod, illustrating schema validation.
- Interactions for code generation and streaming modes where applicable.
- Dependency of LLMScraper on both the LLM provider and the Playwright framework.

##### Architectural Patterns and Design Principles
- Modular design: separate directories and files indicate clear modular boundaries (core functionality in src/ and examples that act as integration tests or demos).
- Abstraction: the LLM provider abstraction supports multiple services.
- Type-safety and schema validation: built-in through TypeScript and Zod, ensuring reliable data structures across the system.
- Extensibility: support for adding new providers, formatting modes (html, markdown, text, image), and code generation capabilities.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Application Layer
    subgraph "Client Application"
        CA1["Codegen Example"]:::client
        CA2["HackerNews Example"]:::client
        CA3["Local Example"]:::client
        CA4["Ollama Example"]:::client
        CA5["Streaming Example"]:::client
    end

    %% Browser Automation Layer
    subgraph "Browser Automation"
        PB["Playwright Browser"]:::external
    end

    %% Core Library Layer
    subgraph "LLMScraper Core"
        LLM1["Main Implementation"]:::core
        LLM2["Models & Schema Validation"]:::core
        LLM3["Cleanup Tasks"]:::core
        LLM4["Code Generation Component"]:::optional
    end

    %% LLM Provider Layer
    subgraph "LLM Provider"
        LLMprov["Provider Integration (OpenAI/Ollama/GGUF)"]:::external
    end

    %% Relationships
    %% Client Application to Browser Automation and Core Library
    CA1 -->|"launches"| PB
    CA2 -->|"launches"| PB
    CA3 -->|"launches"| PB
    CA4 -->|"launches"| PB
    CA5 -->|"launches"| PB

    CA1 -->|"initiates"| LLM1
    CA2 -->|"initiates"| LLM1
    CA3 -->|"initiates"| LLM1
    CA4 -->|"initiates"| LLM1
    CA5 -->|"initiates"| LLM1

    %% Core Library internal interactions
    LLM1 -->|"invokes"| LLMprov
    LLM1 -->|"performs"| LLM3
    LLM1 -->|"utilizes"| LLM2
    LLM1 -->|"optionally generates"| LLM4

    %% Code Generation output back to client
    LLM4 -->|"returns-code"| CA1

    %% Styling
    classDef client fill:#FAD7A0,stroke:#D35400,stroke-width:2px;
    classDef core fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef external fill:#A9DFBF,stroke:#145A32,stroke-width:2px;
    classDef optional fill:#F9E79F,stroke:#B9770E,stroke-width:2px,stroke-dasharray: 5,5;

    %% Click Events
    click CA1 "https://github.com/mishushakov/llm-scraper/blob/main/examples/codegen.ts"
    click CA2 "https://github.com/mishushakov/llm-scraper/blob/main/examples/hn.ts"
    click CA3 "https://github.com/mishushakov/llm-scraper/blob/main/examples/local.ts"
    click CA4 "https://github.com/mishushakov/llm-scraper/blob/main/examples/ollama.ts"
    click CA5 "https://github.com/mishushakov/llm-scraper/blob/main/examples/streaming.ts"
    click LLM1 "https://github.com/mishushakov/llm-scraper/blob/main/src/index.ts"
    click LLM2 "https://github.com/mishushakov/llm-scraper/blob/main/src/models.ts"
    click LLM3 "https://github.com/mishushakov/llm-scraper/blob/main/src/cleanup.ts"
    click LLM4 "https://github.com/mishushakov/llm-scraper/blob/main/examples/codegen.ts"
    click LLMprov "https://github.com/mishushakov/llm-scraper/blob/main/examples/ollama.ts"
    click LLM2 "https://github.com/mishushakov/llm-scraper/blob/main/src/models.ts"
```
