# tech-engine/goscrapy architecture

> Architecture diagram of the GitHub repository tech-engine/goscrapy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tech-engine/goscrapy
- Repository: https://github.com/tech-engine/goscrapy

## Overview

##### Project Type and Purpose
- GoScrapy is an open-source web-scraping framework written in Go, inspired by Python’s Scrapy.
- It provides a CLI for project scaffolding and a set of pluggable components—core spider, engine, scheduler, executor, middleware manager, pipeline manager, adapters, built-in middlewares/pipelines—for building spiders that fetch, parse, and store data.

##### Main Architectural Layers & Components
A. CLI Layer (cmd/cli, cmd/gos)
- `goscrapy startproject` (templates folder) – generates project skeleton (constants.go, errors.go, job.go, record.go, settings.go, spider.go, main.go).
- `gos` command (cmd/gos) – entrypoint for running a spider (gos.New… core).
B. Core API Layer (pkg/core)
- Defines ICoreSpider interface, request/response types, and options for clients, middlewares, pipelines.
C. Engine & Coordination Layer (pkg/engine, internal/fsm, internal/resource_pool, internal/cmap)
- Engine orchestrates the spider lifecycle: Start, Stop, error handling.
- FSM & concurrent resource pool manage spider states and worker goroutines.
- Concurrent map (`cmap`) for shared state or deduplication.
D. Scheduling Layer (pkg/scheduler)
- Scheduler accepts new Requests, queues them, and dispatches to workers.
- Selectors (e.g., FIFO/LIFO), request/response types, worker lifecycle.
E. Execution Layer (pkg/executor + pkg/executor_adapters/http_native)
- Executor interface for fetching HTTP.
- Adapter (http_native) implements HTTP client calls, handles retries, cookie jars, proxies.
F. Middleware Layer (pkg/middlewaremanager + pkg/builtin/middlewares)
- MiddlewareManager applies request/response middlewares (dupefilter, retry, multi_cookiejar).
G. Pipeline Layer (pkg/pipeline_manager + pkg/builtin/pipelines)
- PipelineManager collects items yielded by spider callbacks.
- Built-in exporters: CSV, JSON, MongoDB, Firebase, Google Sheets, dummy.

##### Component Interactions & Data Flow
1. User’s Spider (in generated project) calls `gos.New[*Record]()` → returns core spider instance.
2. CoreSpider.Start(ctx) invokes Engine.Start:
- Reactor initializes Scheduler, Executor, MiddlewareManager, PipelineManager.
- Initial Request(s) from StartRequest() are fed into Scheduler.
3. Scheduler dispatches Request → Worker → Executor.
4. Executor uses HTTP adapter to perform GET/POST → returns Response.
5. Middleware chain wraps request/response before/after network call.
6. Response passed back to spider’s parse callback → yields Record items.
7. PipelineManager routes items through configured pipelines → e.g., export_to_csv.
8. Errors propagate via an error channel returned by NewSpider.

##### Architectural Patterns & Principles
- Plugin architecture: clear ports (interfaces) and adapters; core modules depend on ports only.
- Concurrency via worker pool and resource pool.
- Separation of concerns: scheduling, execution, middleware, pipelines all decoupled.
- CLI scaffolding separated from library code.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Project Layer
    subgraph "User Project Code" 
        direction TB
        UserSpider["User Spider\n(Generated Code)"]:::user
    end

    %% Core API Layer
    subgraph "Core API Layer"
        direction TB
        CoreCore["core.go"]:::core
        CorePorts["ports.go"]:::core
    end

    %% Engine & Coordination Layer
    subgraph "Engine & Coordination Layer"
        direction TB
        EngineCore["engine.go"]:::core
        EnginePorts["ports.go"]:::core
        subgraph "Internal Support"
            direction TB
            FSM["fsm.go"]:::internal
            PoolBuilder["pool_builder.go"]:::internal
            ResourcePool["resource_pool.go"]:::internal
            CMap1["cmap.go"]:::internal
            CMapH["cmaph.go"]:::internal
            CMapTypes["types.go"]:::internal
        end
    end

    %% Scheduling Layer
    subgraph "Scheduling Layer"
        direction TB
        SchedulerCore["scheduler.go"]:::core
        SchedulerWorker["worker.go"]:::core
        SchedulerPorts["ports.go"]:::core
    end

    %% Execution Layer
    subgraph "Execution Layer"
        direction TB
        ExecutorInt["executor.go"]:::core
        ExecutorPorts["ports.go"]:::core
        subgraph "HTTP Adapter"
            direction TB
            HTTPAdapter["adapter.go"]:::adapter
            HTTPHelper["helper.go"]:::adapter
        end
    end

    %% Middleware Layer
    subgraph "Middleware Layer"
        direction TB
        MiddlewareMgr["middlewaremanager.go"]:::core
        subgraph "Built-in Middlewares"
            direction TB
            DupeFilter["dupefilter.go"]:::adapter
            Retry["retry.go"]:::adapter
            MultiCookie["multi_cookiejar.go"]:::adapter
        end
    end

    %% Pipeline Layer
    subgraph "Pipeline Layer"
        direction TB
        PipelineMgr["pipeline_manager.go"]:::core
        subgraph "Built-in Pipelines"
            direction TB
            Dummy["dummy.go"]:::adapter
            ExportCSV["export_to_csv.go"]:::adapter
            ExportJSON["export_to_json.go"]:::adapter
        end
    end

    %% External System
    ExternalSite["Target Website"]:::external

    %% Connections
    UserSpider -->|"uses"| CoreCore
    CorePorts -->|implements| EngineCore
    EngineCore -->|orchestrates| SchedulerCore
    SchedulerWorker -->|dispatches| HTTPAdapter
    HTTPAdapter -->|fetches| ExternalSite
    ExternalSite -->|response| HTTPAdapter
    HTTPAdapter -->|response| EngineCore
    EngineCore -->|feeds| SchedulerCore
    SchedulerCore -->|requests| ExecutorInt
    ExecutorInt -->|raw calls| HTTPAdapter
    HTTPAdapter -->|wrapped by| Retry
    Retry -->|dupe filter| DupeFilter
    DupeFilter -->|cookie jar| MultiCookie
    EngineCore -->|passes items| PipelineMgr
    PipelineMgr -->|dummy| Dummy
    PipelineMgr -->|csv| ExportCSV
    PipelineMgr -->|json| ExportJSON

    %% Click Events
    click CoreCore "https://github.com/tech-engine/goscrapy/blob/main/pkg/core/core.go"
    click CorePorts "https://github.com/tech-engine/goscrapy/blob/main/pkg/core/ports.go"
    click EngineCore "https://github.com/tech-engine/goscrapy/blob/main/pkg/engine/engine.go"
    click EnginePorts "https://github.com/tech-engine/goscrapy/blob/main/pkg/engine/ports.go"
    click FSM "https://github.com/tech-engine/goscrapy/blob/main/internal/fsm/fsm.go"
    click PoolBuilder "https://github.com/tech-engine/goscrapy/blob/main/internal/resource_pool/pool_builder.go"
    click ResourcePool "https://github.com/tech-engine/goscrapy/blob/main/internal/resource_pool/resource_pool.go"
    click CMap1 "https://github.com/tech-engine/goscrapy/blob/main/internal/cmap/cmap.go"
    click CMapH "https://github.com/tech-engine/goscrapy/blob/main/internal/cmap/cmaph.go"
    click CMapTypes "https://github.com/tech-engine/goscrapy/blob/main/internal/cmap/types.go"
    click SchedulerCore "https://github.com/tech-engine/goscrapy/blob/main/pkg/scheduler/scheduler.go"
    click SchedulerWorker "https://github.com/tech-engine/goscrapy/blob/main/pkg/scheduler/worker.go"
    click SchedulerPorts "https://github.com/tech-engine/goscrapy/blob/main/pkg/scheduler/ports.go"
    click ExecutorInt "https://github.com/tech-engine/goscrapy/blob/main/pkg/executor/executor.go"
    click ExecutorPorts "https://github.com/tech-engine/goscrapy/blob/main/pkg/executor/ports.go"
    click HTTPAdapter "https://github.com/tech-engine/goscrapy/blob/main/pkg/executor_adapters/http_native/adapter.go"
    click HTTPHelper "https://github.com/tech-engine/goscrapy/blob/main/pkg/executor_adapters/http_native/helper.go"
    click MiddlewareMgr "https://github.com/tech-engine/goscrapy/blob/main/pkg/middlewaremanager/middlewaremanager.go"
    click DupeFilter "https://github.com/tech-engine/goscrapy/blob/main/pkg/builtin/middlewares/dupefilter.go"
    click Retry "https://github.com/tech-engine/goscrapy/blob/main/pkg/builtin/middlewares/retry.go"
    click MultiCookie "https://github.com/tech-engine/goscrapy/blob/main/pkg/builtin/middlewares/multi_cookiejar.go"
    click PipelineMgr "https://github.com/tech-engine/goscrapy/blob/main/pkg/pipeline_manager/pipeline_manager.go"
    click Dummy "https://github.com/tech-engine/goscrapy/blob/main/pkg/builtin/pipelines/dummy.go"
    click ExportCSV "https://github.com/tech-engine/goscrapy/blob/main/pkg/builtin/pipelines/export_to_csv.go"
    click ExportJSON "https://github.com/tech-engine/goscrapy/blob/main/pkg/builtin/pipelines/export_to_json.go"

    %% Styles
    classDef core fill:#cce5ff,stroke:#004085,color:#004085
    classDef adapter fill:#d4edda,stroke:#155724,color:#155724
    classDef internal fill:#e2e3e5,stroke:#6c757d,color:#6c757d
    classDef user fill:#ffe5b4,stroke:#e69500,color:#e69500
    classDef external fill:#f8d7da,stroke:#721c24,color:#721c24
```
