# postlight/parser architecture

> Architecture diagram of the GitHub repository postlight/parser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/postlight/parser
- Repository: https://github.com/postlight/parser

## Overview

##### Project type and purpose
- This is an open-source JavaScript library & CLI tool (“Postlight Parser”) for extracting article metadata and content from arbitrary web pages.
- It ships as an npm package (`@postlight/parser`) and a command-line binary (`postlight-parser`), with both browser and Node.js support.

A. Client Interfaces
- Browser API (via `<script>` / ES module)
- Node.js API (import `Parser.parse()`)
- CLI binary (`cli.js`)
B. Core Parser Module (`src/mercury.js`)
- Requester / Fetcher (in `src/resource/utils/fetch-resource.js`)
- HTML Parsing & Shims (Cheerio, iconv-lite shims)
- Extractor Manager (`src/extractors/get-extractor.js`)
- Extractors
– Generic extractors (`src/extractors/generic/...`)
– Custom/site-specific extractors (`src/extractors/custom/<domain>`)
- Cleaners (`src/cleaners/...`)
- Formatter (HTML/Markdown/Text output)
C. Build & Bundling
- Rollup configs (`rollup.config.*.js`)
- Dist output (`dist/*.js`, UMD, ESM, web builds)
D. Support Scripts & Fixtures
- Test fixtures (`fixtures/`) & nock mocks
- Automation scripts (`scripts/`)
E. Tests & CI
- Unit tests alongside modules
- Karma for browser tests
- CircleCI config

1. CLI or API → Parser.parse(url or html string, options)
2. Parser → Fetcher → HTTP GET (or uses provided HTML)
3. Raw HTML → Cheerio DOM → Extractor Manager
- Manager picks a custom extractor if domain matches, else falls back to generic extractors
4. Extractor → returns raw fields (title, content, author, date, images, next-page URLs, etc.)
5. Cleaners pipeline → normalize/clean each field (e.g., strip junk tags, resolve split titles)
6. Formatter → produces final HTML, Markdown, or plain-text output
7. Result → returned to caller or printed to STDOUT in CLI

##### Architectural patterns and principles
- Modular pipeline: clear separation of concerns (fetch → extract → clean → format)
- Strategy pattern for extractors: plug-in custom logic per domain, generic fallback
- Layered architecture:
- Resource Layer (HTTP, DOM parsing)
- Extraction Layer (generic vs. custom)
- Cleaning Layer
- Presentation Layer (formatting)
- Build-time bundling to support multiple targets (Node UMD, ESM, browser)

##### Technologies & frameworks
- Runtime: Node.js, Browser
- HTTP: `node-fetch`, `nock` for mocks
- DOM parsing: `cheerio` with shims
- Bundler: Rollup (with configurations for ESM & web)
- Testing: Jest/Mocha style tests, Karma for browser, fixtures & nock
- CLI: `commander` or native argv parsing in `cli.js`

##### Essential elements
- Arrowheads showing data flow from client → parser → fetcher → extractor → cleaner → formatter → output

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client Interfaces
    subgraph "Client Interfaces"
        BrowserAPI["Browser API"]:::runtime
        NodeAPI["Node.js API"]:::runtime
        CLIBinary["CLI Binary"]:::runtime
    end

    %% Core Parser Module
    subgraph "Core Parser Module"
        Parser["Parser (Parser.parse)"]:::runtime
        Fetcher["Requester / Fetcher"]:::runtime
        ShimsCheerio["Cheerio Shim"]:::runtime
        ShimsIconv["Iconv-lite Shim"]:::runtime
        ExtractorMgr["Extractor Manager"]:::runtime
        GenericExt["Generic Extractors"]:::runtime
        CustomExt["Custom Extractors"]:::runtime
        Cleaners["Cleaners"]:::runtime
        FormatterText["Formatter (Text)"]:::runtime
        FormatterDOM["Formatter (HTML/Markdown)"]:::runtime
    end

    %% Build & Bundling
    subgraph "Build & Bundling"
        RollupCommon["rollup.config.js"]:::build
        RollupESM["rollup.config.esm.js"]:::build
        RollupWeb["rollup.config.web.js"]:::build
        DistUMD["dist/mercury.js"]:::build
        DistESM["dist/mercury.esm.js"]:::build
        DistWeb["dist/mercury.web.js"]:::build
    end

    %% Support Scripts & Fixtures
    subgraph "Support Scripts & Fixtures"
        Fixtures["Test Fixtures"]:::test
        HTTPMocks["HTTP Mocks (nock)"]:::test
        Scripts["Support Scripts"]:::build
    end

    %% Tests & CI
    subgraph "Tests & CI"
        UnitTests["Unit Tests"]:::test
        Karma["Karma Config"]:::test
        CI["CircleCI Config"]:::test
    end

    %% Data Flow
    BrowserAPI -->|"calls parse()"| Parser
    NodeAPI -->|"calls parse()"| Parser
    CLIBinary -->|"invokes parse()"| Parser
    Parser -->|"uses"| Fetcher
    Fetcher -->|"HTTP GET → HTML"| ShimsCheerio
    ShimsCheerio -->|"DOM parsing"| ShimsIconv
    ShimsIconv -->|"shims applied"| ExtractorMgr
    ExtractorMgr -->|"select extractor"| GenericExt
    ExtractorMgr -->|"select extractor"| CustomExt
    GenericExt -->|"extract → raw fields"| Cleaners
    CustomExt -->|"extract → raw fields"| Cleaners
    Cleaners -->|"clean fields"| FormatterText
    Cleaners -->|"clean fields"| FormatterDOM
    FormatterText -->|"formatted output"| CLIBinary
    FormatterDOM -->|"formatted output"| CLIBinary

    %% Build pipeline
    RollupCommon --> RollupESM
    RollupCommon --> RollupWeb
    RollupESM --> DistESM
    RollupCommon --> DistUMD
    RollupWeb --> DistWeb

    %% Support & Tests
    Fixtures --> UnitTests
    HTTPMocks --> UnitTests
    Scripts --> UnitTests
    Karma --> Fixtures
    CI --> UnitTests

    %% Click Events
    click BrowserAPI "https://github.com/postlight/parser/blob/main/dist/mercury.web.js"
    click NodeAPI "https://github.com/postlight/parser/blob/main/src/mercury.js"
    click CLIBinary "https://github.com/postlight/parser/blob/main/cli.js"
    click Parser "https://github.com/postlight/parser/blob/main/src/mercury.js"
    click Fetcher "https://github.com/postlight/parser/blob/main/src/resource/utils/fetch-resource.js"
    click ShimsCheerio "https://github.com/postlight/parser/blob/main/src/shims/cheerio-query.js"
    click ShimsIconv "https://github.com/postlight/parser/blob/main/src/shims/iconv-lite.js"
    click ExtractorMgr "https://github.com/postlight/parser/blob/main/src/extractors/get-extractor.js"
    click GenericExt "https://github.com/postlight/parser/tree/main/src/extractors/generic/"
    click CustomExt "https://github.com/postlight/parser/tree/main/src/extractors/custom/"
    click Cleaners "https://github.com/postlight/parser/tree/main/src/cleaners/"
    click FormatterText "https://github.com/postlight/parser/tree/main/src/utils/text/"
    click FormatterDOM "https://github.com/postlight/parser/tree/main/src/utils/dom/"
    click RollupCommon "https://github.com/postlight/parser/blob/main/rollup.config.js"
    click RollupESM "https://github.com/postlight/parser/blob/main/rollup.config.esm.js"
    click RollupWeb "https://github.com/postlight/parser/blob/main/rollup.config.web.js"
    click DistUMD "https://github.com/postlight/parser/blob/main/dist/mercury.js"
    click DistESM "https://github.com/postlight/parser/blob/main/dist/mercury.esm.js"
    click DistWeb "https://github.com/postlight/parser/blob/main/dist/mercury.web.js"
    click Fixtures "https://github.com/postlight/parser/tree/main/fixtures/"
    click HTTPMocks "https://github.com/postlight/parser/tree/main/fixtures/nock/"
    click Scripts "https://github.com/postlight/parser/tree/main/scripts/"
    click Karma "https://github.com/postlight/parser/blob/main/karma.conf.js"
    click CI "https://github.com/postlight/parser/blob/main/.circleci/config.yml"

    %% Styles
    classDef runtime fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef build fill:#d2f8d2,stroke:#333,stroke-width:1px
    classDef test fill:#f0f0f0,stroke:#333,stroke-width:1px
```
