How builderio/gpt-crawler works
Project type and purpose
• This project is a specialized web crawler (“GPT Crawler”) intended to traverse one or more websites, extract content based on configurable selectors, and output a JSON file that can be uploaded to OpenAI for use in creating custom GPTs or custom assistants.
• The README explains how to run it using Node.js, within containers, and as an API (via Express), so the architecture supports multiple run modes.
File structure
• Top-level files and directories indicate several aspects:
• Configuration and environment files (config.ts, .env.example) allow global configuration.
• A “src” directory holds the core application logic (files like cli.ts, core.ts, main.ts, server.ts) – the crawler logic and the Express server code are located here.
• The “containerapp” directory includes Docker-related files (its own Dockerfile, run.sh, a README, and a config.ts in a data subdirectory) for containerized execution.
• Auxiliary files such as swagger.js suggest that API documentation (with swagger) is generated, and the presence of a Dockerfile in the root indicates container support.
• These pieces suggest that the application can operate in different environments (CLI mode vs. server mode) and is built with a modular design where configuration, crawling logic, and deployment are separated.
Components
a. Main Components:
• CLI Entry Point – represented by “src/cli.ts” and “src/main.ts” which handle local execution.
• API Server – “src/server.ts” runs an Express server to process crawl requests (note endpoints like “/crawl” and the API docs at “/api-docs” served by swagger).
• Crawler Core Logic – “src/core.ts” encapsulates the core functionality of crawling, data extraction (using selectors and matching URLs), and output generation.
• Configuration Module – The configuration files (“config.ts” at the root and in “containerapp/data/config.ts”) determine the crawling behavior and data extraction options.
• Docker/Container Support – The Dockerfiles (both at the root and within containerapp) and container-specific scripts (run.sh) show that containerization is supported, providing portability.
• External Integration – The generated “output.json” is intended to be used by OpenAI services (custom GPT or assistant creation). There’s also documentation via swagger and integration with GitHub workflows.
b. Relationships and Interactions:
• In CLI mode, the execution flow starts when the user runs the CLI entry point. The crawler core (core.ts) consumes configuration (from config.ts) and performs crawling, then writes its results to a JSON file.
• When running as an API, an Express server (server.ts) exposes endpoints (particularly the “/crawl” endpoint) that receive configuration parameters (or use defaults) via a POST request. The server then calls the crawler core, and the response (i.e., data generation) is returned to the client.
• In containerized mode, the container’s entry point (e.g., run.sh in containerapp) configures the environment, reads configuration, starts the crawler (or even the API server), and mounts/outputs data to/from the container.
• Swagger’s integration (referenced in swagger.js) documents the API endpoints and is accessible via the /api-docs endpoint when the server is running.
• The configuration files serve as the bridge between various modes and allow customization of crawling criteria such as URL, selector, and page limits.
c. Important Architectural Patterns and Technologies:
• Modular design separating configuration, crawling logic, and API serving.
• Dual-mode execution (CLI and HTTP API) with shared core crawling functionality.
• Use of a configuration-as-code principle (editable config.ts files).
• Containerization via Docker, ensuring portability and consistent deployments.
• API documentation provision via Swagger.
• Built with Node.js (using Express for the server mode) and configured through package.json and tsconfig.json.
• Entry Point: “src/cli.ts” or “src/main.ts” → calls → “src/core.ts” → writes to → “output.json”
• Configuration used from “config.ts”
• HTTP Request to “/crawl” and “/api-docs” handled by “src/server.ts”
• The server then calls “src/core.ts” for crawling.
• Swagger module connects to provide API documentation.
• A block for the containerized deployment:
• “Dockerfile” and “containerapp/run.sh” encapsulate the application.
• That container configuration (e.g., environment from .env.example, containerapp/data/config.ts) feeds into the same crawler core.
• Configuration → Crawler Core → Output file.
• API Request → Server → Crawler Core → Response/Output file.
• Application logic (source files in “src”)
• Configuration artifacts (config.ts files, .env)
• Infrastructure components (Docker, API, CLI)
• External services (OpenAI endpoints for uploading the output)