# crawlab-team/crawlab architecture

> Architecture diagram of the GitHub repository crawlab-team/crawlab, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/crawlab-team/crawlab
- Repository: https://github.com/crawlab-team/crawlab

## Overview

Crawlab is a distributed crawler-management platform: users manage spiders and schedules through a web UI and master-hosted API; the master schedules tasks to workers, which run crawler processes and store logs and results. The architecture also includes node management, authentication, statistics, exports, notifications, and configurable data sources. Relationships are grounded in the README and the supplied source index or excerpts; some unsampled subsystem internals are shown only at the documented level.

Main entry points: [`core/node/service/master_service_v2.go`](https://github.com/crawlab-team/crawlab/blob/main/core/node/service/master_service_v2.go) (Master Node), [`core/controllers/router_v2.go`](https://github.com/crawlab-team/crawlab/blob/main/core/controllers/router_v2.go) (HTTP API), [`core/task/handler/runner_v2.go`](https://github.com/crawlab-team/crawlab/blob/main/core/task/handler/runner_v2.go) (Task Runner), [`core/task/scheduler/service_v2.go`](https://github.com/crawlab-team/crawlab/blob/main/core/task/scheduler/service_v2.go) (Task Scheduler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui_api["UI and API"]
  node_frontend["Web Frontend<br/>[main.ts]"]
  node_api["HTTP API<br/>[router_v2.go]"]
  node_auth["Authentication<br/>[service_v2.go]"]
end

subgraph group_control["Master Control"]
  node_master["Master Node"]
  node_spiders["Spider Management<br/>[service.go]"]
  node_schedules["Schedules<br/>[service_v2.go]"]
  node_scheduler["Task Scheduler<br/>[service_v2.go]"]
  node_grpc_server["gRPC Server<br/>[server_v2.go]"]
end

subgraph group_execution["Task Execution"]
  node_worker["Worker Node"]
  node_grpc_client["gRPC Client<br/>[client.go]"]
  node_handler["Task Handler<br/>[service_v2.go]"]
  node_runner["Task Runner<br/>[runner_v2.go]"]
end

subgraph group_data["Data and Files"]
  node_models["Model Services<br/>[base_service.go]"]
  node_mongo[("MongoDB")]
  node_files["Distributed Files"]
  node_results["Result Storage<br/>[service_mongo.go]"]
end

subgraph group_insights["Insights and Integrations"]
  node_stats["Statistics<br/>[service.go]"]
  node_exports["CSV Export<br/>[csv_service.go]"]
  node_notifications["Notifications<br/>[service_v2.go]"]
  node_data_sources["Data Sources<br/>[service.go]"]
end

node_user(("Platform User"))

node_user -->|"uses"| node_frontend
node_frontend -->|"calls"| node_api
node_api -.->|"authenticates"| node_auth
node_api -.->|"serves requests"| node_models
node_master -->|"starts"| node_grpc_server
node_master -->|"uses"| node_schedules
node_master -->|"schedules tasks"| node_scheduler
node_master -->|"deploys spiders"| node_spiders
node_spiders -->|"uploads files"| node_files
node_schedules -.->|"triggers tasks"| node_scheduler
node_scheduler -->|"dispatches tasks"| node_worker
node_grpc_server -->|"communicates"| node_grpc_client
node_worker -->|"uses"| node_grpc_client
node_grpc_client -.->|"delivers tasks"| node_handler
node_handler -->|"runs tasks"| node_runner
node_runner -->|"stores logs"| node_files
node_runner -->|"saves results"| node_results
node_models -->|"reads and writes"| node_mongo
node_results -->|"writes results"| node_mongo
node_stats -->|"queries metrics"| node_mongo
node_exports -->|"exports records"| node_mongo
node_master -.->|"sends alerts"| node_notifications
node_data_sources -->|"checks and saves"| node_models

click node_frontend "https://github.com/crawlab-team/crawlab/blob/main/frontend/src/main.ts"
click node_api "https://github.com/crawlab-team/crawlab/blob/main/core/controllers/router_v2.go"
click node_auth "https://github.com/crawlab-team/crawlab/blob/main/core/user/service_v2.go"
click node_master "https://github.com/crawlab-team/crawlab/blob/main/core/node/service/master_service_v2.go"
click node_spiders "https://github.com/crawlab-team/crawlab/blob/main/core/spider/admin/service.go"
click node_schedules "https://github.com/crawlab-team/crawlab/blob/main/core/schedule/service_v2.go"
click node_scheduler "https://github.com/crawlab-team/crawlab/blob/main/core/task/scheduler/service_v2.go"
click node_grpc_server "https://github.com/crawlab-team/crawlab/blob/main/core/grpc/server/server_v2.go"
click node_worker "https://github.com/crawlab-team/crawlab/blob/main/core/node/service/worker_service_v2.go"
click node_grpc_client "https://github.com/crawlab-team/crawlab/blob/main/core/grpc/client/client.go"
click node_handler "https://github.com/crawlab-team/crawlab/blob/main/core/task/handler/service_v2.go"
click node_runner "https://github.com/crawlab-team/crawlab/blob/main/core/task/handler/runner_v2.go"
click node_models "https://github.com/crawlab-team/crawlab/blob/main/core/models/service/base_service.go"
click node_files "https://github.com/crawlab-team/crawlab/blob/main/fs/seaweedfs_manager.go"
click node_results "https://github.com/crawlab-team/crawlab/blob/main/core/result/service_mongo.go"
click node_stats "https://github.com/crawlab-team/crawlab/blob/main/core/stats/service.go"
click node_exports "https://github.com/crawlab-team/crawlab/blob/main/core/export/csv_service.go"
click node_notifications "https://github.com/crawlab-team/crawlab/blob/main/core/notification/service_v2.go"
click node_data_sources "https://github.com/crawlab-team/crawlab/blob/main/core/ds/service.go"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend,node_api,node_auth,node_user toneBlue
class node_master,node_spiders,node_schedules,node_scheduler,node_grpc_server toneAmber
class node_worker,node_grpc_client,node_handler,node_runner toneMint
class node_models,node_mongo,node_files,node_results toneRose
class node_stats,node_exports,node_notifications,node_data_sources toneIndigo
```

## Components

### UI and API

- [Web Frontend](https://github.com/crawlab-team/crawlab/blob/main/frontend/src/main.ts): `frontend/src/main.ts`
- [HTTP API](https://github.com/crawlab-team/crawlab/blob/main/core/controllers/router_v2.go): `core/controllers/router_v2.go`
- [Authentication](https://github.com/crawlab-team/crawlab/blob/main/core/user/service_v2.go): `core/user/service_v2.go`

### Master Control

- [Master Node](https://github.com/crawlab-team/crawlab/blob/main/core/node/service/master_service_v2.go): `core/node/service/master_service_v2.go`
- [Spider Management](https://github.com/crawlab-team/crawlab/blob/main/core/spider/admin/service.go): `core/spider/admin/service.go`
- [Schedules](https://github.com/crawlab-team/crawlab/blob/main/core/schedule/service_v2.go): `core/schedule/service_v2.go`
- [Task Scheduler](https://github.com/crawlab-team/crawlab/blob/main/core/task/scheduler/service_v2.go): `core/task/scheduler/service_v2.go`
- [gRPC Server](https://github.com/crawlab-team/crawlab/blob/main/core/grpc/server/server_v2.go): `core/grpc/server/server_v2.go`

### Task Execution

- [Worker Node](https://github.com/crawlab-team/crawlab/blob/main/core/node/service/worker_service_v2.go): `core/node/service/worker_service_v2.go`
- [gRPC Client](https://github.com/crawlab-team/crawlab/blob/main/core/grpc/client/client.go): `core/grpc/client/client.go`
- [Task Handler](https://github.com/crawlab-team/crawlab/blob/main/core/task/handler/service_v2.go): `core/task/handler/service_v2.go`
- [Task Runner](https://github.com/crawlab-team/crawlab/blob/main/core/task/handler/runner_v2.go): `core/task/handler/runner_v2.go`

### Data and Files

- [Model Services](https://github.com/crawlab-team/crawlab/blob/main/core/models/service/base_service.go): `core/models/service/base_service.go`
- **MongoDB**
- [Distributed Files](https://github.com/crawlab-team/crawlab/blob/main/fs/seaweedfs_manager.go): `fs/seaweedfs_manager.go`
- [Result Storage](https://github.com/crawlab-team/crawlab/blob/main/core/result/service_mongo.go): `core/result/service_mongo.go`

### Insights and Integrations

- [Statistics](https://github.com/crawlab-team/crawlab/blob/main/core/stats/service.go): `core/stats/service.go`
- [CSV Export](https://github.com/crawlab-team/crawlab/blob/main/core/export/csv_service.go): `core/export/csv_service.go`
- [Notifications](https://github.com/crawlab-team/crawlab/blob/main/core/notification/service_v2.go): `core/notification/service_v2.go`
- [Data Sources](https://github.com/crawlab-team/crawlab/blob/main/core/ds/service.go): `core/ds/service.go`

### Other components

- **Platform User**

## Connections

- Platform User → Web Frontend: uses
- Web Frontend → HTTP API: calls
- HTTP API → Authentication: authenticates
- HTTP API → Model Services: serves requests
- Master Node → gRPC Server: starts (evidence: `core/node/service/master_service_v2.go`)
- Master Node → Schedules: uses (evidence: `core/node/service/master_service_v2.go`)
- Master Node → Task Scheduler: schedules tasks
- Master Node → Spider Management: deploys spiders
- Spider Management → Distributed Files: uploads files
- Schedules → Task Scheduler: triggers tasks
- Task Scheduler → Worker Node: dispatches tasks
- gRPC Server → gRPC Client: communicates
- Worker Node → gRPC Client: uses (evidence: `core/node/service/worker_service_v2.go`)
- gRPC Client → Task Handler: delivers tasks
- Task Handler → Task Runner: runs tasks
- Task Runner → Distributed Files: stores logs
- Task Runner → Result Storage: saves results
- Model Services → MongoDB: reads and writes (evidence: `core/models/service/base_service.go`)
- Result Storage → MongoDB: writes results (evidence: `core/result/service_mongo.go`)
- Statistics → MongoDB: queries metrics (evidence: `core/stats/service.go`)
- CSV Export → MongoDB: exports records (evidence: `core/export/csv_service.go`)
- Master Node → Notifications: sends alerts
- Data Sources → Model Services: checks and saves (evidence: `core/ds/service.go`)
