How spookydooky/beluga works
Beluga is a configurable web-scraping engine exposed through a REST API. The principal workflow is to define a job and its extraction rules, create and execute task runs, fetch pages and extract configured data, then store results for later retrieval. Separate capabilities include job/execution management, activity logging, selectable persistence and filesystem/S3 result storage. The tree and excerpts establish the main runtime stages; unsampled subsystem links are omitted where the available evidence does not show a call or dispatch.
Main entry points: JobExecutorService.java Job Executor · ResultFileService.java Result Service · JobDefinitionService.java Definition Service · JobService.java Job Service
Components 21
REST API
- Job API
src/beluga-scraper/src/main/java/com/beluga/api/job/controller/JobController.java - Execution API
src/beluga-scraper/src/main/java/com/beluga/api/execution/controller/ExecutionController.java - Task API
src/beluga-scraper/src/main/java/com/beluga/api/task/controller/TaskController.java - Results API
src/beluga-scraper/src/main/java/com/beluga/api/results/controller/ResultsController.java
Job Management
- Job Definitions
src/beluga-scraper/src/main/java/com/beluga/model/job_definition/JobDefinition.java - Job Service
src/beluga-scraper/src/main/java/com/beluga/service/job/JobService.java - Definition Service
src/beluga-scraper/src/main/java/com/beluga/service/job/JobDefinitionService.java - Execution Service
src/beluga-scraper/src/main/java/com/beluga/api/execution/service/ExecutionApiService.java
Scraping Runtime
- Job Executor
src/beluga-scraper/src/main/java/com/beluga/execution/service/job/JobExecutorService.java - Task Queue
src/beluga-scraper/src/main/java/com/beluga/execution/service/task/JobTaskQueue.java - Worker
src/beluga-scraper/src/main/java/com/beluga/execution/service/worker/Worker.java - Task Execution
src/beluga-scraper/src/main/java/com/beluga/service/task/TaskExecutionService.java - Scraping Engine
src/beluga-scraper/src/main/java/com/beluga/scraping/ScrapingService.java - HTTP Retrieval
src/beluga-scraper/src/main/java/com/beluga/http/HttpService.java
Persistence and Results
- Job Persistence
src/beluga-scraper/src/main/java/com/beluga/persistence/repository/JobDefinitionRepository.java - Result Storage
src/beluga-scraper/src/main/java/com/beluga/result_storage/ResultStorageService.java - Storage Provider
src/beluga-scraper/src/main/java/com/beluga/result_storage/ResultDataStoreProvider.java - Result Service
src/beluga-scraper/src/main/java/com/beluga/service/results/ResultFileService.java
Activity Logging
- Activity Logging
src/beluga-scraper/src/main/java/com/beluga/activity_logging/service/ActivityLoggingService.java - Activity Repository
src/beluga-scraper/src/main/java/com/beluga/activity_logging/repository/ActivityLogRepository.java
Other components
- API Client
Connections 18
- API Client → Job API: manages jobs
README.md - API Client → Execution API: controls runs
README.md - API Client → Results API: retrieves results
README.md - Job API → Definition Service: delegates to
- Execution API → Execution Service: delegates to
- Execution Service → Job Executor: starts execution
src/beluga-scraper/src/main/java/com/beluga/api/execution/service/ExecutionApiService.java - Job Service → Definition Service: loads definitions
src/beluga-scraper/src/main/java/com/beluga/service/job/JobService.java - Job Service → Task Execution: saves task runs
src/beluga-scraper/src/main/java/com/beluga/service/job/JobService.java - Definition Service → Job Persistence: reads and writes
src/beluga-scraper/src/main/java/com/beluga/service/job/JobDefinitionService.java - Job Executor → Task Queue: polls tasks
src/beluga-scraper/src/main/java/com/beluga/execution/service/job/JobExecutorService.java - Job Executor → Worker: dispatches tasks
src/beluga-scraper/src/main/java/com/beluga/execution/service/job/JobExecutorService.java - Worker → Scraping Engine: runs scraping
src/beluga-scraper/src/main/java/com/beluga/execution/service/worker/Worker.java - Scraping Engine → HTTP Retrieval: retrieves page
src/beluga-scraper/src/main/java/com/beluga/scraping/ScrapingService.java - Results API → Result Service: retrieves files
- Result Storage → Storage Provider: saves and retrieves
src/beluga-scraper/src/main/java/com/beluga/result_storage/ResultStorageService.java - Result Service → Job Persistence: stores file records
src/beluga-scraper/src/main/java/com/beluga/service/results/ResultFileService.java - Result Service → Result Storage: uses stored data
src/beluga-scraper/src/main/java/com/beluga/api/results/service/ResultService.java - Activity Logging → Activity Repository: stores activity