How crawlab-team/crawlab works
Crawlab is a distributed crawler-management platform: users manage spiders and schedules through a web UI and master-hosted API; the master schedules tasks to workers, which run crawler processes and store logs and results. The architecture also includes node management, authentication, statistics, exports, notifications, and configurable data sources. Relationships are grounded in the README and the supplied source index or excerpts; some unsampled subsystem internals are shown only at the documented level.
Main entry points: master_service_v2.go Master Node · router_v2.go HTTP API · runner_v2.go Task Runner · service_v2.go Task Scheduler
Components 21
UI and API
- Web Frontend
frontend/src/main.ts - HTTP API
core/controllers/router_v2.go - Authentication
core/user/service_v2.go
Master Control
- Master Node
core/node/service/master_service_v2.go - Spider Management
core/spider/admin/service.go - Schedules
core/schedule/service_v2.go - Task Scheduler
core/task/scheduler/service_v2.go - gRPC Server
core/grpc/server/server_v2.go
Task Execution
- Worker Node
core/node/service/worker_service_v2.go - gRPC Client
core/grpc/client/client.go - Task Handler
core/task/handler/service_v2.go - Task Runner
core/task/handler/runner_v2.go
Data and Files
- Model Services
core/models/service/base_service.go - MongoDB
- Distributed Files
fs/seaweedfs_manager.go - Result Storage
core/result/service_mongo.go
Insights and Integrations
- Statistics
core/stats/service.go - CSV Export
core/export/csv_service.go - Notifications
core/notification/service_v2.go - Data Sources
core/ds/service.go
Other components
- Platform User
Connections 23
- Platform User → Web Frontend: uses
- Web Frontend → HTTP API: calls
- HTTP API → Authentication: authenticates
- HTTP API → Model Services: serves requests
- Master Node → gRPC Server: starts
core/node/service/master_service_v2.go - Master Node → Schedules: uses
core/node/service/master_service_v2.go - Master Node → Task Scheduler: schedules tasks
- Master Node → Spider Management: deploys spiders
- Spider Management → Distributed Files: uploads files
- Schedules → Task Scheduler: triggers tasks
- Task Scheduler → Worker Node: dispatches tasks
- gRPC Server → gRPC Client: communicates
- Worker Node → gRPC Client: uses
core/node/service/worker_service_v2.go - gRPC Client → Task Handler: delivers tasks
- Task Handler → Task Runner: runs tasks
- Task Runner → Distributed Files: stores logs
- Task Runner → Result Storage: saves results
- Model Services → MongoDB: reads and writes
core/models/service/base_service.go - Result Storage → MongoDB: writes results
core/result/service_mongo.go - Statistics → MongoDB: queries metrics
core/stats/service.go - CSV Export → MongoDB: exports records
core/export/csv_service.go - Master Node → Notifications: sends alerts
- Data Sources → Model Services: checks and saves
core/ds/service.go