# builderio/gpt-crawler architecture

> Architecture diagram of the GitHub repository builderio/gpt-crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/builderio/gpt-crawler
- Repository: https://github.com/builderio/gpt-crawler
- Video tour (about a minute): https://gitdiagram.com/builderio/gpt-crawler/video

## Overview

##### Project type and purpose
- This project is a specialized web crawler (“GPT Crawler”) intended to traverse one or more websites, extract content based on configurable selectors, and output a JSON file that can be uploaded to OpenAI for use in creating custom GPTs or custom assistants.
- The README explains how to run it using Node.js, within containers, and as an API (via Express), so the architecture supports multiple run modes.

##### File structure
- Top-level files and directories indicate several aspects:
- Configuration and environment files (config.ts, .env.example) allow global configuration.
- A “src” directory holds the core application logic (files like cli.ts, core.ts, main.ts, server.ts) – the crawler logic and the Express server code are located here.
- The “containerapp” directory includes Docker-related files (its own Dockerfile, run.sh, a README, and a config.ts in a data subdirectory) for containerized execution.
- Auxiliary files such as swagger.js suggest that API documentation (with swagger) is generated, and the presence of a Dockerfile in the root indicates container support.
- These pieces suggest that the application can operate in different environments (CLI mode vs. server mode) and is built with a modular design where configuration, crawling logic, and deployment are separated.

##### Components
a. Main Components:
- CLI Entry Point – represented by “src/cli.ts” and “src/main.ts” which handle local execution.
- API Server – “src/server.ts” runs an Express server to process crawl requests (note endpoints like “/crawl” and the API docs at “/api-docs” served by swagger).
- Crawler Core Logic – “src/core.ts” encapsulates the core functionality of crawling, data extraction (using selectors and matching URLs), and output generation.
- Configuration Module – The configuration files (“config.ts” at the root and in “containerapp/data/config.ts”) determine the crawling behavior and data extraction options.
- Docker/Container Support – The Dockerfiles (both at the root and within containerapp) and container-specific scripts (run.sh) show that containerization is supported, providing portability.
- External Integration – The generated “output.json” is intended to be used by OpenAI services (custom GPT or assistant creation). There’s also documentation via swagger and integration with GitHub workflows.

b. Relationships and Interactions:
- In CLI mode, the execution flow starts when the user runs the CLI entry point. The crawler core (core.ts) consumes configuration (from config.ts) and performs crawling, then writes its results to a JSON file.
- When running as an API, an Express server (server.ts) exposes endpoints (particularly the “/crawl” endpoint) that receive configuration parameters (or use defaults) via a POST request. The server then calls the crawler core, and the response (i.e., data generation) is returned to the client.
- In containerized mode, the container’s entry point (e.g., run.sh in containerapp) configures the environment, reads configuration, starts the crawler (or even the API server), and mounts/outputs data to/from the container.
- Swagger’s integration (referenced in swagger.js) documents the API endpoints and is accessible via the /api-docs endpoint when the server is running.
- The configuration files serve as the bridge between various modes and allow customization of crawling criteria such as URL, selector, and page limits.

c. Important Architectural Patterns and Technologies:
- Modular design separating configuration, crawling logic, and API serving.
- Dual-mode execution (CLI and HTTP API) with shared core crawling functionality.
- Use of a configuration-as-code principle (editable config.ts files).
- Containerization via Docker, ensuring portability and consistent deployments.
- API documentation provision via Swagger.
- Built with Node.js (using Express for the server mode) and configured through package.json and tsconfig.json.

- Entry Point: “src/cli.ts” or “src/main.ts” → calls → “src/core.ts” → writes to → “output.json”
- Configuration used from “config.ts”
- HTTP Request to “/crawl” and “/api-docs” handled by “src/server.ts”
- The server then calls “src/core.ts” for crawling.
- Swagger module connects to provide API documentation.
- A block for the containerized deployment:
- “Dockerfile” and “containerapp/run.sh” encapsulate the application.
- That container configuration (e.g., environment from .env.example, containerapp/data/config.ts) feeds into the same crawler core.
- Configuration → Crawler Core → Output file.
- API Request → Server → Crawler Core → Response/Output file.
- Application logic (source files in “src”)
- Configuration artifacts (config.ts files, .env)
- Infrastructure components (Docker, API, CLI)
- External services (OpenAI endpoints for uploading the output)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Components
    CFG["Configuration (config.ts)"]:::config
    CORE["Crawler Core (src/core.ts)"]:::core
    OUT["Output JSON (output.json)"]:::external
    OPENAI["OpenAI Integration"]:::external

    %% CLI Mode
    subgraph "CLI Mode"
        CLI["CLI Entry Point (src/cli.ts, src/main.ts)"]:::cli
    end

    %% API Mode
    subgraph "API Mode"
        API["API Server (src/server.ts)"]:::api
        SWAG["Swagger Docs (swagger.js)"]:::external
    end

    %% Container Deployment
    subgraph "Container Deployment"
        DOCK["Dockerfile (root)"]:::docker
        CNTDOCK["Container Dockerfile (containerapp/Dockerfile)"]:::docker
        RUN["Container Script (containerapp/run.sh)"]:::docker
        CNTCFG["Container Config (containerapp/data/config.ts)"]:::config
    end

    %% Relationships for CLI Mode
    CLI -->|"calls"| CORE
    CFG -->|"configures"| CORE
    CORE -->|"writes"| OUT

    %% Relationships for API Mode
    API -->|"invokes"| CORE
    CORE -->|"generates"| OUT
    API -->|"serves"| SWAG

    %% Container deployment relationships
    CNTCFG -->|"overrides"| CFG
    RUN -->|"starts"| CLI
    RUN -->|"starts"| API

    %% External integration
    OUT -->|"uploads"| OPENAI

    %% Styles
    classDef cli fill:#ffcccc,stroke:#000,stroke-width:2px;
    classDef api fill:#ccffcc,stroke:#000,stroke-width:2px;
    classDef core fill:#ccccff,stroke:#000,stroke-width:2px;
    classDef config fill:#fff2cc,stroke:#000,stroke-width:2px;
    classDef docker fill:#e0e0e0,stroke:#000,stroke-width:2px;
    classDef external fill:#f0d9ff,stroke:#000,stroke-width:2px;

    %% Click Events
    click CLI "https://github.com/builderio/gpt-crawler/blob/main/src/cli.ts"
    click CLI "https://github.com/builderio/gpt-crawler/blob/main/src/main.ts"
    click API "https://github.com/builderio/gpt-crawler/blob/main/src/server.ts"
    click CORE "https://github.com/builderio/gpt-crawler/blob/main/src/core.ts"
    click CFG "https://github.com/builderio/gpt-crawler/blob/main/config.ts"
    click CNTCFG "https://github.com/builderio/gpt-crawler/blob/main/containerapp/data/config.ts"
    click DOCK "https://github.com/builderio/gpt-crawler/tree/main/Dockerfile"
    click CNTDOCK "https://github.com/builderio/gpt-crawler/tree/main/containerapp/Dockerfile"
    click RUN "https://github.com/builderio/gpt-crawler/blob/main/containerapp/run.sh"
    click SWAG "https://github.com/builderio/gpt-crawler/blob/main/swagger.js"
```
