How tech-engine/goscrapy works
Project Type and Purpose
• GoScrapy is an open-source web-scraping framework written in Go, inspired by Python’s Scrapy.
• It provides a CLI for project scaffolding and a set of pluggable components—core spider, engine, scheduler, executor, middleware manager, pipeline manager, adapters, built-in middlewares/pipelines—for building spiders that fetch, parse, and store data.
Main Architectural Layers & Components
A. CLI Layer (cmd/cli, cmd/gos)
• goscrapy startproject (templates folder) – generates project skeleton (constants.go, errors.go, job.go, record.go, settings.go, spider.go, main.go).
• gos command (cmd/gos) – entrypoint for running a spider (gos.New… core).
B. Core API Layer (pkg/core)
• Defines ICoreSpider interface, request/response types, and options for clients, middlewares, pipelines.
C. Engine & Coordination Layer (pkg/engine, internal/fsm, internal/resource_pool, internal/cmap)
• Engine orchestrates the spider lifecycle: Start, Stop, error handling.
• FSM & concurrent resource pool manage spider states and worker goroutines.
• Concurrent map (cmap) for shared state or deduplication.
D. Scheduling Layer (pkg/scheduler)
• Scheduler accepts new Requests, queues them, and dispatches to workers.
• Selectors (e.g., FIFO/LIFO), request/response types, worker lifecycle.
E. Execution Layer (pkg/executor + pkg/executor_adapters/http_native)
• Executor interface for fetching HTTP.
• Adapter (http_native) implements HTTP client calls, handles retries, cookie jars, proxies.
F. Middleware Layer (pkg/middlewaremanager + pkg/builtin/middlewares)
• MiddlewareManager applies request/response middlewares (dupefilter, retry, multi_cookiejar).
G. Pipeline Layer (pkg/pipeline_manager + pkg/builtin/pipelines)
• PipelineManager collects items yielded by spider callbacks.
• Built-in exporters: CSV, JSON, MongoDB, Firebase, Google Sheets, dummy.
Component Interactions & Data Flow
1. User’s Spider (in generated project) calls gos.New[*Record]() → returns core spider instance.
2. CoreSpider.Start(ctx) invokes Engine.Start:
• Reactor initializes Scheduler, Executor, MiddlewareManager, PipelineManager.
• Initial Request(s) from StartRequest() are fed into Scheduler.
3. Scheduler dispatches Request → Worker → Executor.
4. Executor uses HTTP adapter to perform GET/POST → returns Response.
5. Middleware chain wraps request/response before/after network call.
6. Response passed back to spider’s parse callback → yields Record items.
7. PipelineManager routes items through configured pipelines → e.g., export_to_csv.
8. Errors propagate via an error channel returned by NewSpider.
Architectural Patterns & Principles
• Plugin architecture: clear ports (interfaces) and adapters; core modules depend on ports only.
• Concurrency via worker pool and resource pool.
• Separation of concerns: scheduling, execution, middleware, pipelines all decoupled.
• CLI scaffolding separated from library code.