How mishushakov/llm-scraper works
Project Type and Purpose
• The LLM Scraper is an extraction library that connects a webpage’s content with LLM processing. It is not a full-stack web app but rather a dedicated scraping tool. Its purpose is to abstract the complexity of scraping web pages and converting them into structured, type-safe objects (validated by Zod) using an LLM.
• Key use-cases include scraping, structured data extraction, code generation for scraping scripts, and streaming data extraction.
File Structure
• Top-level files include configuration files (.gitignore, .prettierrc, tsconfig.json, package.json), which confirm this is a Node/TypeScript project.
• The “src” directory contains the core library code (e.g., LLMScraper’s main implementation in index.ts, models.ts, and cleanup.ts).
• The “examples” directory provides usage examples (codegen.ts, hn.ts, local.ts, ollama.ts, streaming.ts) that illustrate different functionalities and providers.
• This clean separation between source code and examples suggests a library design intended for developers to integrate into their projects.
README for Additional Insights
• The README explains how to install and use the scraper with various LLM providers.
• It outlines the different modes of operation (standard scraping, streaming, and code-generation) and demonstrates how the Playwright framework is used to manage browser interactions.
• The examples include multiple integration patterns: initializing LLMs, scraping with defined schemas (using Zod), and generating scraping code.
Main Components
a. Client Application (User Code):
• The consumer’s code that sets up and utilizes the LLM Scraper (as seen in the examples).
• This includes initiating a Playwright browser and passing a page instance to the scraper.
b. LLMScraper Library:
• The core module (found in src/) that handles the overall logic, taking a provided LLM instance to process page content.
• Inside this module, include separate tasks for cleanup, schema validation, and integration with Playwright.
c. LLM Providers:
• Abstraction for various LLM services (OpenAI, Ollama, GGUF, etc.).
• How the user code can pass different providers to the scraper constructor, emphasizing support for multiple platforms.
d. Playwright:
• The browser automation tool used to navigate to webpages. It manages page creation, navigation, and content extraction.
e. Code Generation Component:
• An optional feature within the LLMScraper that generates scraping scripts based on provided schemas.
• This should be shown as an extension of the base functionality.
f. Data Schema Validation:
• Integration with Zod which ensures that the data returned by the LLM meets the expected structured schema defined by the user.
Relationships and Interactions
• From the Client Application to Playwright (launching and managing a browser page).
• From the Client Application to the LLMScraper, showing initialization with an LLM provider.
• From the LLMScraper to the LLM Provider, representing the data extraction using function calling under the hood.
• From the LLMScraper to Zod, illustrating schema validation.
• Interactions for code generation and streaming modes where applicable.
• Dependency of LLMScraper on both the LLM provider and the Playwright framework.
Architectural Patterns and Design Principles
• Modular design: separate directories and files indicate clear modular boundaries (core functionality in src/ and examples that act as integration tests or demos).
• Abstraction: the LLM provider abstraction supports multiple services.
• Type-safety and schema validation: built-in through TypeScript and Zod, ensuring reliable data structures across the system.
• Extensibility: support for adding new providers, formatting modes (html, markdown, text, image), and code generation capabilities.