# spookydooky/beluga architecture

> Architecture diagram of the GitHub repository spookydooky/beluga, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/spookydooky/beluga
- Repository: https://github.com/spookydooky/beluga

## Overview

Beluga is a configurable web-scraping engine exposed through a REST API. The principal workflow is to define a job and its extraction rules, create and execute task runs, fetch pages and extract configured data, then store results for later retrieval. Separate capabilities include job/execution management, activity logging, selectable persistence and filesystem/S3 result storage. The tree and excerpts establish the main runtime stages; unsampled subsystem links are omitted where the available evidence does not show a call or dispatch.

Main entry points: [`src/beluga-scraper/src/main/java/com/beluga/execution/service/job/JobExecutorService.java`](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/execution/service/job/JobExecutorService.java) (Job Executor), [`src/beluga-scraper/src/main/java/com/beluga/service/results/ResultFileService.java`](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/service/results/ResultFileService.java) (Result Service), [`src/beluga-scraper/src/main/java/com/beluga/service/job/JobDefinitionService.java`](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/service/job/JobDefinitionService.java) (Definition Service), [`src/beluga-scraper/src/main/java/com/beluga/service/job/JobService.java`](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/service/job/JobService.java) (Job Service)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["REST API"]
  node_job_api["Job API<br/>[JobController.java]"]
  node_execution_api["Execution API"]
  node_task_api["Task API"]
  node_results_api["Results API"]
end

subgraph group_jobs["Job Management"]
  node_job_definitions["Job Definitions<br/>[JobDefinition.java]"]
  node_job_service["Job Service<br/>[JobService.java]"]
  node_job_definition_service["Definition Service"]
  node_execution_service["Execution Service"]
end

subgraph group_runtime["Scraping Runtime"]
  node_job_executor["Job Executor"]
  node_task_queue["Task Queue<br/>[JobTaskQueue.java]"]
  node_worker["Worker<br/>[Worker.java]"]
  node_task_execution_service["Task Execution"]
  node_scraper["Scraping Engine"]
  node_http["HTTP Retrieval<br/>[HttpService.java]"]
end

subgraph group_storage["Persistence and Results"]
  node_persistence[("Job Persistence")]
  node_result_storage["Result Storage"]
  node_storage_provider["Storage Provider"]
  node_result_service["Result Service"]
end

subgraph group_observability["Activity Logging"]
  node_activity_service["Activity Logging"]
  node_activity_repository[("Activity Repository")]
end

node_client(("API Client"))

node_client -->|"manages jobs"| node_job_api
node_client -->|"controls runs"| node_execution_api
node_client -->|"retrieves results"| node_results_api
node_job_api -.->|"delegates to"| node_job_definition_service
node_execution_api -.->|"delegates to"| node_execution_service
node_execution_service -->|"starts execution"| node_job_executor
node_job_service -->|"loads definitions"| node_job_definition_service
node_job_service -->|"saves task runs"| node_task_execution_service
node_job_definition_service -->|"reads and writes"| node_persistence
node_job_executor -->|"polls tasks"| node_task_queue
node_job_executor -->|"dispatches tasks"| node_worker
node_worker -.->|"runs scraping"| node_scraper
node_scraper -->|"retrieves page"| node_http
node_results_api -.->|"retrieves files"| node_result_service
node_result_storage -->|"saves and retrieves"| node_storage_provider
node_result_service -->|"stores file records"| node_persistence
node_result_service -->|"uses stored data"| node_result_storage
node_activity_service -.->|"stores activity"| node_activity_repository

click node_job_api "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/api/job/controller/JobController.java"
click node_execution_api "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/api/execution/controller/ExecutionController.java"
click node_task_api "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/api/task/controller/TaskController.java"
click node_results_api "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/api/results/controller/ResultsController.java"
click node_job_definitions "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/model/job_definition/JobDefinition.java"
click node_job_service "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/service/job/JobService.java"
click node_job_definition_service "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/service/job/JobDefinitionService.java"
click node_execution_service "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/api/execution/service/ExecutionApiService.java"
click node_job_executor "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/execution/service/job/JobExecutorService.java"
click node_task_queue "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/execution/service/task/JobTaskQueue.java"
click node_worker "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/execution/service/worker/Worker.java"
click node_task_execution_service "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/service/task/TaskExecutionService.java"
click node_scraper "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/scraping/ScrapingService.java"
click node_http "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/http/HttpService.java"
click node_persistence "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/persistence/repository/JobDefinitionRepository.java"
click node_result_storage "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/result_storage/ResultStorageService.java"
click node_storage_provider "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/result_storage/ResultDataStoreProvider.java"
click node_result_service "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/service/results/ResultFileService.java"
click node_activity_service "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/activity_logging/service/ActivityLoggingService.java"
click node_activity_repository "https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/activity_logging/repository/ActivityLogRepository.java"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_job_api,node_execution_api,node_task_api,node_results_api,node_client toneBlue
class node_job_definitions,node_job_service,node_job_definition_service,node_execution_service toneAmber
class node_job_executor,node_task_queue,node_worker,node_task_execution_service,node_scraper,node_http toneMint
class node_persistence,node_result_storage,node_storage_provider,node_result_service toneRose
class node_activity_service,node_activity_repository toneIndigo
```

## Components

### REST API

- [Job API](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/api/job/controller/JobController.java): `src/beluga-scraper/src/main/java/com/beluga/api/job/controller/JobController.java`
- [Execution API](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/api/execution/controller/ExecutionController.java): `src/beluga-scraper/src/main/java/com/beluga/api/execution/controller/ExecutionController.java`
- [Task API](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/api/task/controller/TaskController.java): `src/beluga-scraper/src/main/java/com/beluga/api/task/controller/TaskController.java`
- [Results API](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/api/results/controller/ResultsController.java): `src/beluga-scraper/src/main/java/com/beluga/api/results/controller/ResultsController.java`

### Job Management

- [Job Definitions](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/model/job_definition/JobDefinition.java): `src/beluga-scraper/src/main/java/com/beluga/model/job_definition/JobDefinition.java`
- [Job Service](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/service/job/JobService.java): `src/beluga-scraper/src/main/java/com/beluga/service/job/JobService.java`
- [Definition Service](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/service/job/JobDefinitionService.java): `src/beluga-scraper/src/main/java/com/beluga/service/job/JobDefinitionService.java`
- [Execution Service](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/api/execution/service/ExecutionApiService.java): `src/beluga-scraper/src/main/java/com/beluga/api/execution/service/ExecutionApiService.java`

### Scraping Runtime

- [Job Executor](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/execution/service/job/JobExecutorService.java): `src/beluga-scraper/src/main/java/com/beluga/execution/service/job/JobExecutorService.java`
- [Task Queue](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/execution/service/task/JobTaskQueue.java): `src/beluga-scraper/src/main/java/com/beluga/execution/service/task/JobTaskQueue.java`
- [Worker](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/execution/service/worker/Worker.java): `src/beluga-scraper/src/main/java/com/beluga/execution/service/worker/Worker.java`
- [Task Execution](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/service/task/TaskExecutionService.java): `src/beluga-scraper/src/main/java/com/beluga/service/task/TaskExecutionService.java`
- [Scraping Engine](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/scraping/ScrapingService.java): `src/beluga-scraper/src/main/java/com/beluga/scraping/ScrapingService.java`
- [HTTP Retrieval](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/http/HttpService.java): `src/beluga-scraper/src/main/java/com/beluga/http/HttpService.java`

### Persistence and Results

- [Job Persistence](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/persistence/repository/JobDefinitionRepository.java): `src/beluga-scraper/src/main/java/com/beluga/persistence/repository/JobDefinitionRepository.java`
- [Result Storage](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/result_storage/ResultStorageService.java): `src/beluga-scraper/src/main/java/com/beluga/result_storage/ResultStorageService.java`
- [Storage Provider](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/result_storage/ResultDataStoreProvider.java): `src/beluga-scraper/src/main/java/com/beluga/result_storage/ResultDataStoreProvider.java`
- [Result Service](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/service/results/ResultFileService.java): `src/beluga-scraper/src/main/java/com/beluga/service/results/ResultFileService.java`

### Activity Logging

- [Activity Logging](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/activity_logging/service/ActivityLoggingService.java): `src/beluga-scraper/src/main/java/com/beluga/activity_logging/service/ActivityLoggingService.java`
- [Activity Repository](https://github.com/spookydooky/beluga/blob/main/src/beluga-scraper/src/main/java/com/beluga/activity_logging/repository/ActivityLogRepository.java): `src/beluga-scraper/src/main/java/com/beluga/activity_logging/repository/ActivityLogRepository.java`

### Other components

- **API Client**

## Connections

- API Client → Job API: manages jobs (evidence: `README.md`)
- API Client → Execution API: controls runs (evidence: `README.md`)
- API Client → Results API: retrieves results (evidence: `README.md`)
- Job API → Definition Service: delegates to
- Execution API → Execution Service: delegates to
- Execution Service → Job Executor: starts execution (evidence: `src/beluga-scraper/src/main/java/com/beluga/api/execution/service/ExecutionApiService.java`)
- Job Service → Definition Service: loads definitions (evidence: `src/beluga-scraper/src/main/java/com/beluga/service/job/JobService.java`)
- Job Service → Task Execution: saves task runs (evidence: `src/beluga-scraper/src/main/java/com/beluga/service/job/JobService.java`)
- Definition Service → Job Persistence: reads and writes (evidence: `src/beluga-scraper/src/main/java/com/beluga/service/job/JobDefinitionService.java`)
- Job Executor → Task Queue: polls tasks (evidence: `src/beluga-scraper/src/main/java/com/beluga/execution/service/job/JobExecutorService.java`)
- Job Executor → Worker: dispatches tasks (evidence: `src/beluga-scraper/src/main/java/com/beluga/execution/service/job/JobExecutorService.java`)
- Worker → Scraping Engine: runs scraping (evidence: `src/beluga-scraper/src/main/java/com/beluga/execution/service/worker/Worker.java`)
- Scraping Engine → HTTP Retrieval: retrieves page (evidence: `src/beluga-scraper/src/main/java/com/beluga/scraping/ScrapingService.java`)
- Results API → Result Service: retrieves files
- Result Storage → Storage Provider: saves and retrieves (evidence: `src/beluga-scraper/src/main/java/com/beluga/result_storage/ResultStorageService.java`)
- Result Service → Job Persistence: stores file records (evidence: `src/beluga-scraper/src/main/java/com/beluga/service/results/ResultFileService.java`)
- Result Service → Result Storage: uses stored data (evidence: `src/beluga-scraper/src/main/java/com/beluga/api/results/service/ResultService.java`)
- Activity Logging → Activity Repository: stores activity
