How web-agent-master/google-search works
Project Type and Purpose
• This project is a Playwright-based Node.js tool designed to perform Google searches locally while bypassing anti-scraping mechanisms. It serves as both a command-line utility and an MCP server that can integrate with AI assistants (such as Claude).
• Its key focus is on browser automation (using Playwright) and robust search result extraction, along with a server component for real-time search capabilities.
Analyzing the File Structure
• At the top level, you have configuration files (package.json, tsconfig.json), build artifacts (bin for executables), and the src/ directory that contains the core TypeScript code.
• The src/ folder includes:
• index.ts (the main entry point for command-line usage and the initial parsing logic)
• search.ts (contains the logic responsible for handling the browser automation to perform searches)
• mcp-server.ts (implements the MCP server functionality for AI assistant integration)
• logger.ts and types.ts (for logging and type definitions)
• The bin directory contains executable scripts, which indicate that the project is designed to be used as a command-line tool (with both standard search and MCP-specific entry points).
Architectural Components
• Command-Line Interface (CLI):
• Entry point (index.ts) which handles argument parsing (likely using Commander) and triggers search functionalities.
• Search Module:
• Search.ts component that manages browser automation via Playwright.
• Interactions with the logger for detailed output and error handling.
• MCP Server:
• Mcp-server.ts component which stands apart from the CLI, providing a long-running server process to serve search queries in real-time.
• Integration with external AI assistants (e.g., Claude) using the Model Context Protocol.
• Shared Utilities:
• Supporting modules such as logger.ts for logging and types.ts for shared type definitions.
• Underlying Infrastructure:
• Node.js runtime environment, TypeScript transpilation (indicated by tsconfig.json and package.json), and dependency management via pnpm.
• Playwright for browser automation across different modes (headless/headed) with state management for anti-bot detection.
Relationships and Data Flow
• For MCP functionality, illustrate the mcp-server.ts component receiving external requests via a standardized protocol (MCP) and then internally calling the search functionality.
• Error handling flows where errors from the browser automation and network issues are captured and logged via logger.ts.
Architectural Patterns and Design Principles
• The separation of concerns: clearly divide the application into a CLI part vs. a server component.
• Encapsulation of browser automation in the search module, ensuring that complexities like fingerprint management, state saving, and error recovery are modularized.
• Integration via a protocol (MCP), which isolates the search service from the client (e.g., Claude).
• Use of TypeScript for type safety and structured coding with domain-specific types in types.ts.
Including Relevant Technologies
• Node.js as the execution environment.
• TypeScript as the development language.
• Playwright for browser interaction.
• Commander (or similar) for command line parsing.
• MCP SDK and Zod for protocol implementation and schema validation.
• pnpm for package management.
• Deployment aspects—mention that the tool uses compiled JavaScript (via tsconfig.json) and that executable scripts reside in the bin/ directory.
• Error handling and logging as a cross-cutting concern (show that errors from search.ts and mcp-server.ts are handed off to the logger).
• Use annotations to note special handling for Windows environments (like .cmd files and cross-platform path handling), which might affect deployment or runtime behavior.