# j-hoplin/puppeteer-pool-manager architecture

> Architecture diagram of the GitHub repository j-hoplin/puppeteer-pool-manager, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/j-hoplin/puppeteer-pool-manager
- Repository: https://github.com/j-hoplin/puppeteer-pool-manager

## Overview

##### Project Type and Purpose
- Type: A Node.js/TypeScript library (npm package) providing a high-level API to manage a pool of Puppeteer contexts.
- Purpose: Enable concurrent, resource-efficient web scraping/automation by recycling browser contexts, managing task queues (default or priority), monitoring resource usage (CPU/memory), and exposing metrics.

##### File Structure
Top-level areas in `src/`:
- client/ — the public, static API surface (start, runTask, stop, getPoolMetrics)
- configs/ — config loader & validator for pool, context, threshold
- decorator/ — injection of state or behavior into methods or classes
- error/ — validation and error-handling logic for configs, pool state, tasks
- logger/ — logging abstraction (levels, enable/disable)
- pool/
– context/ — implementations for ISOLATED vs SHARED Puppeteer contexts
– dispatcher.ts — orchestrates pulling tasks from queue and assigning to contexts
– enum.ts — enums for ContextMode, LogLevel, QueueMode
– index.ts — core PoolManager class (singleton)
- queue/ — default FIFO queue and priority queue implementations
- watcher/ — threshold watcher (CPU/memory polling, safe pool reset)
- types/ — shared type definitions

##### README Insights
- Core functionality: `PuppeteerPool.start()`, `.runTask()`, `.stop()`, `.getPoolMetrics()`.
- Configurable: concurrencyLevel, contextMode, taskQueueType, custom configs, logging.
- Features: priority task queue, context GC, threshold watcher with auto-reset, metrics reporting.

##### Main Components and Their Interactions

A. Client API Layer (src/client)
– Exposed static methods → use cases (start, runTask, stop, getMetrics)

B. Configuration Manager (src/configs)
– Loads `puppeteer-pool-config.json` or defaults, validates settings
– Injects into PoolManager on start.

C. PoolManager (singleton) (src/pool/index.ts)
– Holds pool state, pool contexts, dispatcher, queue, watcher, logger
– On .start(): initializes contexts & dispatcher
– On .stop(): terminates contexts

D. Task Queue (src/queue)
– DefaultQueue (FIFO) or PriorityQueue (heap) per user option
– Provides `enqueue(task, priority?)` and `dequeue()`

E. Dispatcher (src/pool/dispatcher.ts)
– Listens for new tasks, pulls from queue, assigns to available context
– Manages concurrencyLevel

F. Context Factory & Manager (src/pool/context)
– Two modes: SHARED (reuse pages within a single browser instance) or ISOLATED (new browser/page per task)
– Creates Puppeteer browser/page, runs task callback, handles GC

G. Threshold Watcher (src/watcher)
– Polls CPU & memory every interval, triggers PoolManager reset if thresholds exceeded

H. Metrics Collector (src/watcher/metrics.ts or integrated in PoolManager)
– Gathers pool metrics: memoryUsage, cpuUsage, sessionCount

I. Error Handling & Validation (src/error + src/decorator/state.ts)
– Validates inputs, catches errors in tasks or context lifecycle, propagates errors to client

J. Logger (src/logger)
– Uniform logging for events, errors, watcher alerts

K. External Dependency: Puppeteer Library
– Launches browser instances per context

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Layer
    User["User Application"]:::external
    User -->|"calls start(), runTask(), stop(), getMetrics()"| ClientAPI

    %% Client API Layer
    subgraph "Client API Layer"
        direction TB
        APIIndex["index.ts"]:::publicapi
        APICluster["cluster.ts"]:::publicapi
        APIRest["rest.ts"]:::publicapi
    end
    User --> ClientAPI(("Client API")):::publicapi
    ClientAPI --> APIIndex
    ClientAPI --> APICluster
    ClientAPI --> APIRest

    %% Configuration Manager
    subgraph "Configuration Manager"
        direction TB
        ConfigIndex["index.ts"]:::internal
        ConfigLoader["config.ts"]:::internal
        ConfigValidator["validator.ts"]:::internal
    end
    ClientAPI -->|"loads configs"| ConfigurationManager(("Config Manager")):::internal
    ConfigurationManager --> ConfigIndex
    ConfigurationManager --> ConfigLoader
    ConfigurationManager --> ConfigValidator

    %% Core PoolManager
    subgraph "Core PoolManager"
        direction TB
        PoolEnum["enum.ts"]:::internal
        PoolManager["index.ts"]:::internal
        Dispatcher["dispatcher.ts"]:::internal
        subgraph "Task Queue"
            direction TB
            QueueIndex["index.ts"]:::internal
            QueueBase["queue.ts"]:::internal
            QueuePriority["queue.priority.ts"]:::internal
            QueueInterface["queue.interface.ts"]:::internal
        end
        subgraph "Context Factory & Manager"
            direction TB
            ContextIndex["index.ts"]:::internal
            ContextBase["context.ts"]:::internal
            ContextIsolate["isolate.ts"]:::internal
            ContextShared["shared.ts"]:::internal
        end
    end
    ConfigurationManager --> PoolManager
    ClientAPI --> PoolManager
    PoolManager --> PoolEnum
    PoolManager --> Dispatcher
    PoolManager -->|"uses queue"| QueueIndex
    Dispatcher --> QueueIndex
    Dispatcher -->|"assigns tasks to"| ContextIndex
    PoolManager --> ContextIndex

    %% Support Modules
    subgraph "Support Modules"
        direction TB
        Watcher["index.ts"]:::internal
        Metrics["metrics.ts"]:::internal
        subgraph "Error Handling & Validation"
            direction TB
            ErrorIndex["index.ts"]:::internal
            ErrorPool["pool.ts"]:::internal
            ErrorState["state.ts"]:::internal
            ErrorValidator["validator.ts"]:::internal
            DecoratorState["state.ts"]:::internal
        end
        Logger["index.ts"]:::internal
        TypesIndex["types/index.ts"]:::internal
        TypesType["types/type.ts"]:::internal
    end
    PoolManager --> Watcher
    Watcher --> Metrics
    PoolManager --> Logger
    PoolManager --> ErrorIndex
    ErrorIndex --> ErrorPool
    ErrorIndex --> ErrorState
    ErrorIndex --> ErrorValidator
    ErrorState --> DecoratorState
    PoolManager --> TypesIndex
    TypesIndex --> TypesType

    %% External Dependency
    Puppeteer["Puppeteer Library"]:::external
    ContextBase --> Puppeteer
    ContextIsolate --> Puppeteer
    ContextShared --> Puppeteer

    %% Click Events
    click APIIndex "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/client/index.ts"
    click APICluster "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/client/cluster.ts"
    click APIRest "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/client/rest.ts"
    click ConfigIndex "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/configs/index.ts"
    click ConfigLoader "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/configs/config.ts"
    click ConfigValidator "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/configs/validator.ts"
    click PoolManager "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/pool/index.ts"
    click Dispatcher "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/pool/dispatcher.ts"
    click QueueIndex "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/queue/index.ts"
    click QueueBase "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/queue/queue.ts"
    click QueuePriority "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/queue/queue.priority.ts"
    click QueueInterface "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/queue/queue.interface.ts"
    click ContextIndex "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/pool/context/index.ts"
    click ContextBase "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/pool/context/context.ts"
    click ContextIsolate "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/pool/context/isolate.ts"
    click ContextShared "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/pool/context/shared.ts"
    click Watcher "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/watcher/index.ts"
    click Metrics "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/watcher/metrics.ts"
    click ErrorIndex "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/error/index.ts"
    click ErrorPool "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/error/pool.ts"
    click ErrorState "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/error/state.ts"
    click ErrorValidator "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/error/validator.ts"
    click DecoratorState "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/decorator/state.ts"
    click Logger "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/logger/index.ts"
    click TypesIndex "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/types/index.ts"
    click TypesType "https://github.com/j-hoplin/puppeteer-pool-manager/blob/main/src/types/type.ts"

    %% Styles
    classDef publicapi fill:#f9f,stroke:#333,stroke-width:1px
    classDef internal fill:#afa,stroke:#333,stroke-width:1px
    classDef external fill:#ff9,stroke:#333,stroke-width:1px
```
