How mend3/puppeteer-di-solid-ts works
Project type and purpose
• This is a specialized command-line web crawler built in TypeScript using Puppeteer.
• It follows SOLID design principles to remain modular, testable, and maintainable.
• It supports both local browser launches and remote browser connections (via a Docker-based browserless service).
File structure
• Top-level files:
• package.json / package-lock.json — Node.js project metadata and dependencies
• docker-compose.yaml — orchestration for a remote browser service (browserless)
• tsconfig.json — TypeScript compiler options
• .gitignore, LICENSE, ABOUT.md, README.md — documentation and config
• src/puppeteer.ts — the single entry point for the crawler’s logic
• The minimal directory layout and reliance on SOLID suggests the code is organized in distinct classes/modules inside puppeteer.ts (or imported there) for launching browsers, navigating pages, extracting data, etc.
README for additional insights
• Configuration via environment variables for REMOTE_BROWSER and EXECUTABLE_PATH
• npm scripts:
• start — run with local Chrome
• start:remote — pull up browserless via Docker compose and connect
• build — generate docs and produce a dist folder
• Core technologies: Puppeteer, TypeScript, Docker
Main components
a. Internal Components
• Crawler Engine (core orchestrator)
• Browser Launcher Module
– Local Launcher (executes /usr/bin/google-chrome or provided path)
– Remote Launcher (connects to browserless host:port)
• Page Processor (navigation, DOM evaluation, data extraction)
• Configuration Manager (reads env vars, sets exec path or remote host)
• Build & Docs Generator (npm build step, TypeDoc output)
b. External Services
• Local Chrome Browser
• Browserless Docker Service (defined in docker-compose.yaml)
• File System / Dist folder (output artifacts, logs, docs)
c. Relationships and data flows
• Configuration Manager feeds settings into Browser Launcher
• Crawler Engine invokes Browser Launcher → obtains Puppeteer Browser instance
• Browser instance passed to Page Processor → extracts data → writes to FS or forwards to consumer
• Build step is separate—reads src, generates docs, outputs to dist
Important architectural patterns
• SOLID modularity: separate classes for each responsibility
• Dependency injection in TypeScript: inject Launcher into Engine, inject Config into Launcher
• Strategy pattern for Local vs. Remote launcher
• Single entry point (puppeteer.ts) that wires everything together
Detailed element checklist
• Components clearly labeled (e.g., “LocalLauncher,” “RemoteLauncher,” “CrawlerEngine”)
• Data flows annotated (e.g., “Page HTML,” “Config settings”)