How postlight/parser works
Project type and purpose
• This is an open-source JavaScript library & CLI tool (“Postlight Parser”) for extracting article metadata and content from arbitrary web pages.
• It ships as an npm package (@postlight/parser) and a command-line binary (postlight-parser), with both browser and Node.js support.
A. Client Interfaces
• Browser API (via <script> / ES module)
• Node.js API (import Parser.parse())
• CLI binary (cli.js)
B. Core Parser Module (src/mercury.js)
• Requester / Fetcher (in src/resource/utils/fetch-resource.js)
• HTML Parsing & Shims (Cheerio, iconv-lite shims)
• Extractor Manager (src/extractors/get-extractor.js)
• Extractors
– Generic extractors (src/extractors/generic/...)
– Custom/site-specific extractors (src/extractors/custom/<domain>)
• Cleaners (src/cleaners/...)
• Formatter (HTML/Markdown/Text output)
C. Build & Bundling
• Rollup configs (rollup.config.*.js)
• Dist output (dist/*.js, UMD, ESM, web builds)
D. Support Scripts & Fixtures
• Test fixtures (fixtures/) & nock mocks
• Automation scripts (scripts/)
E. Tests & CI
• Unit tests alongside modules
• Karma for browser tests
• CircleCI config
1. CLI or API → Parser.parse(url or html string, options)
2. Parser → Fetcher → HTTP GET (or uses provided HTML)
3. Raw HTML → Cheerio DOM → Extractor Manager
• Manager picks a custom extractor if domain matches, else falls back to generic extractors
4. Extractor → returns raw fields (title, content, author, date, images, next-page URLs, etc.)
5. Cleaners pipeline → normalize/clean each field (e.g., strip junk tags, resolve split titles)
6. Formatter → produces final HTML, Markdown, or plain-text output
7. Result → returned to caller or printed to STDOUT in CLI
Architectural patterns and principles
• Modular pipeline: clear separation of concerns (fetch → extract → clean → format)
• Strategy pattern for extractors: plug-in custom logic per domain, generic fallback
• Layered architecture:
• Resource Layer (HTTP, DOM parsing)
• Extraction Layer (generic vs. custom)
• Cleaning Layer
• Presentation Layer (formatting)
• Build-time bundling to support multiple targets (Node UMD, ESM, browser)
Technologies & frameworks
• Runtime: Node.js, Browser
• HTTP: node-fetch, nock for mocks
• DOM parsing: cheerio with shims
• Bundler: Rollup (with configurations for ESM & web)
• Testing: Jest/Mocha style tests, Karma for browser, fixtures & nock
• CLI: commander or native argv parsing in cli.js
Essential elements
• Arrowheads showing data flow from client → parser → fetcher → extractor → cleaner → formatter → output