How j-hoplin/puppeteer-pool-manager works
Project Type and Purpose
• Type: A Node.js/TypeScript library (npm package) providing a high-level API to manage a pool of Puppeteer contexts.
• Purpose: Enable concurrent, resource-efficient web scraping/automation by recycling browser contexts, managing task queues (default or priority), monitoring resource usage (CPU/memory), and exposing metrics.
File Structure
Top-level areas in src/:
• client/ — the public, static API surface (start, runTask, stop, getPoolMetrics)
• configs/ — config loader & validator for pool, context, threshold
• decorator/ — injection of state or behavior into methods or classes
• error/ — validation and error-handling logic for configs, pool state, tasks
• logger/ — logging abstraction (levels, enable/disable)
• pool/
– context/ — implementations for ISOLATED vs SHARED Puppeteer contexts
– dispatcher.ts — orchestrates pulling tasks from queue and assigning to contexts
– enum.ts — enums for ContextMode, LogLevel, QueueMode
– index.ts — core PoolManager class (singleton)
• queue/ — default FIFO queue and priority queue implementations
• watcher/ — threshold watcher (CPU/memory polling, safe pool reset)
• types/ — shared type definitions
README Insights
• Core functionality: PuppeteerPool.start(), .runTask(), .stop(), .getPoolMetrics().
• Configurable: concurrencyLevel, contextMode, taskQueueType, custom configs, logging.
• Features: priority task queue, context GC, threshold watcher with auto-reset, metrics reporting.
Main Components and Their Interactions
A. Client API Layer (src/client)
– Exposed static methods → use cases (start, runTask, stop, getMetrics)
B. Configuration Manager (src/configs)
– Loads puppeteer-pool-config.json or defaults, validates settings
– Injects into PoolManager on start.
C. PoolManager (singleton) (src/pool/index.ts)
– Holds pool state, pool contexts, dispatcher, queue, watcher, logger
– On .start(): initializes contexts & dispatcher
– On .stop(): terminates contexts
D. Task Queue (src/queue)
– DefaultQueue (FIFO) or PriorityQueue (heap) per user option
– Provides enqueue(task, priority?) and dequeue()
E. Dispatcher (src/pool/dispatcher.ts)
– Listens for new tasks, pulls from queue, assigns to available context
– Manages concurrencyLevel
F. Context Factory & Manager (src/pool/context)
– Two modes: SHARED (reuse pages within a single browser instance) or ISOLATED (new browser/page per task)
– Creates Puppeteer browser/page, runs task callback, handles GC
G. Threshold Watcher (src/watcher)
– Polls CPU & memory every interval, triggers PoolManager reset if thresholds exceeded
H. Metrics Collector (src/watcher/metrics.ts or integrated in PoolManager)
– Gathers pool metrics: memoryUsage, cpuUsage, sessionCount
I. Error Handling & Validation (src/error + src/decorator/state.ts)
– Validates inputs, catches errors in tasks or context lifecycle, propagates errors to client
J. Logger (src/logger)
– Uniform logging for events, errors, watcher alerts
K. External Dependency: Puppeteer Library
– Launches browser instances per context