# romansky/dom-to-semantic-markdown architecture

> Architecture diagram of the GitHub repository romansky/dom-to-semantic-markdown, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/romansky/dom-to-semantic-markdown
- Repository: https://github.com/romansky/dom-to-semantic-markdown

## Overview

##### Project Type and Purpose
- This is an open-source JavaScript/TypeScript library (dom-to-semantic-markdown) that converts HTML DOM into a semantic Markdown AST/string optimized for LLM consumption.
- It provides three primary “faces”:
- A core conversion engine (reusable API)
- A CLI wrapper for fetching HTML (via HTTP or Playwright), calling the core, and writing Markdown
- A browser-side API (for in-page conversion)

##### File Structure Analysis
- Top-Level Config & Metadata:
- .github/workflows → CI (test, lint) and publish pipelines
- package.json, rollup.config.js → bundling/browser+node builds, npm publish
- jest.config.js, tsconfig.*.json → test/build configs
- src/
- src/index.ts → public API exports and CLI entry
- src/core/ → core engine modules (AST utils, DOM traversal, HTML→AST, AST→string, URL utils)
- src/types/ → shared TypeScript types (SemanticMarkdownAST, ConversionOptions)
- tests/ → unit tests for core functionality
- examples/ → sample CLI, browser, and Node.js usage
- issue.p, failing-test.p → repro test cases

##### Main Architectural Components
A. CLI Layer
- Entry script (src/index.ts) invoked by `bin/d2m`.
- Parses flags (–url, –input, –extract-main, –track-table-columns, –include-meta-data, –use-playwright).
- If URL: either HTTP fetch or Playwright fetch → raw HTML.
- Calls Core API → returns Markdown string → writes to stdout/file.
B. Core Conversion Engine
- HTML parsing → DOM (browser DOMParser or jsdom)
- htmlToMarkdownAST (src/core/htmlToMarkdownAST.ts) → SemanticMarkdownAST[]
- markdownASTToString (src/core/markdownASTToString.ts) → Markdown text
- astUtils, domUtils, urlUtils aid transformations, metadata extraction, URL refification.
- Extensibility via overrideElementProcessing, processUnhandledElement, overrideNodeRenderer, renderCustomNode options.
C. Browser and Node API
- Browser: convertHtmlToMarkdown(document.body, options) → Markdown
- Node: convertHtmlToMarkdown(string HTML, options + overrideDOMParser via jsdom)
D. Build & Publish Pipeline
- Rollup builds: outputs CJS/ESM bundles for Node and browser (tsconfigs for each)
- GitHub Actions:
- CI workflow → runs jest tests, lint, typecheck
- Publish workflow → on npm release tags

##### Interactions and Data Flows
- CLI Input → (HTTP / Playwright) → HTML → Core Engine → Markdown → CLI Output
- Browser API: DOM Element → Core Engine → Markdown
- Node API: HTML String → jsdom DOM → Core Engine → Markdown
- Core Engine publishes AST → can be intercepted by custom renderers/plugins

##### Architectural Patterns & Principles
- Separation of Concerns: CLI, core logic, bundling, examples, and tests all live in separate folders
- Plugin-style overrides: dependency injection for element processing/rendering
- Build-time code splitting: separate browser and Node bundles via Rollup/tsconfig
- Test-driven: jest tests for each core feature (metadata extraction, main content detection, table tracking)

##### Technologies & Frameworks
- TypeScript, Rollup, Jest
- Node.js, ECMAScript modules, npm CLI
- jsdom (Node DOM parsing), optional Playwright (dynamic fetch)

## Diagram (Mermaid)

```mermaid
flowchart TB
  subgraph "API Layer"
    CLI["d2m CLI\n(src/index.ts)"]:::runtime
    BrowserAPI["Browser Conversion API\n(convertHtmlToMarkdown)"]:::runtime
    NodeAPI["Node Conversion API\n(+ jsdom)"]:::runtime
  end

  subgraph "Core Engine"
    HTMLtoAST["htmlToMarkdownAST\n(src/core/htmlToMarkdownAST.ts)"]:::runtime
    ASTtoString["markdownASTToString\n(src/core/markdownASTToString.ts)"]:::runtime
    ASTUtils["AST Utilities\n(src/core/astUtils.ts)"]:::runtime
    DOMUtils["DOM Utilities\n(src/core/domUtils.ts)"]:::runtime
    URLUtils["URL Utilities\n(src/core/urlUtils.ts)"]:::runtime
    ElementNode["Element Node Definitions\n(src/core/ElementNode.ts)"]:::runtime
    Types["Shared Types\n(src/types/markdownTypes.ts)"]:::runtime
  end

  subgraph "External Services"
    HTTPFetch["HTTP Fetch"]:::external
    Playwright["Playwright Fetch"]:::external
    npmRegistry["npm Registry"]:::external
  end

  subgraph "Build & CI"
    Rollup["Rollup Bundler\n(rollup.config.js)"]:::build
    TSBrowser["tsconfig.browser.json"]:::build
    TSNode["tsconfig.node.json"]:::build
    CI["CI Workflow\n(.github/workflows/ci.yml)"]:::build
    Publish["Publish Workflow\n(.github/workflows/publish.yml)"]:::build
    Tests["Unit Tests\n(tests/)"]:::build
  end

  subgraph "Examples"
    CLIExample["CLI Example\n(examples/cli)"]:::runtime
    BrowserExample["Browser Example\n(examples/browser.html)"]:::runtime
    NodeExample["Node Example\n(examples/node)"]:::runtime
  end

  CLI -->|"fetch via HTTP"| HTTPFetch
  CLI -->|"fetch via Playwright"| Playwright
  HTTPFetch -->|HTML| HTMLtoAST
  Playwright -->|HTML| HTMLtoAST

  BrowserAPI -->|DOM Element| HTMLtoAST
  NodeAPI -->|HTML String| HTMLtoAST

  HTMLtoAST -->|calls| ASTUtils
  HTMLtoAST -->|calls| DOMUtils
  HTMLtoAST -->|calls| URLUtils
  HTMLtoAST -->|uses| ElementNode
  HTMLtoAST -->|produces| Types
  HTMLtoAST -->|outputs AST| ASTtoString
  ASTtoString -->|produces Markdown| CLI

  CLIExample --> CLI
  BrowserExample --> BrowserAPI
  NodeExample --> NodeAPI

  TSBrowser --> Rollup
  TSNode --> Rollup
  Rollup -->|bundles| npmRegistry
  CI --> Tests
  CI --> TSBrowser
  CI --> TSNode
  CI --> Rollup
  Publish -->|releases to| npmRegistry

  classDef runtime fill:#D0E8FF,stroke:#0366d6,color:#0366d6
  classDef external fill:#DFF3CC,stroke:#22863a,color:#22863a
  classDef build fill:#FFE4B3,stroke:#bf8700,color:#bf8700

  click CLI "https://github.com/romansky/dom-to-semantic-markdown/blob/main/src/index.ts"
  click BrowserAPI "https://github.com/romansky/dom-to-semantic-markdown/blob/main/src/index.ts"
  click NodeAPI "https://github.com/romansky/dom-to-semantic-markdown/blob/main/src/index.ts"
  click HTMLtoAST "https://github.com/romansky/dom-to-semantic-markdown/blob/main/src/core/htmlToMarkdownAST.ts"
  click ASTtoString "https://github.com/romansky/dom-to-semantic-markdown/blob/main/src/core/markdownASTToString.ts"
  click ASTUtils "https://github.com/romansky/dom-to-semantic-markdown/blob/main/src/core/astUtils.ts"
  click DOMUtils "https://github.com/romansky/dom-to-semantic-markdown/blob/main/src/core/domUtils.ts"
  click URLUtils "https://github.com/romansky/dom-to-semantic-markdown/blob/main/src/core/urlUtils.ts"
  click ElementNode "https://github.com/romansky/dom-to-semantic-markdown/blob/main/src/core/ElementNode.ts"
  click Types "https://github.com/romansky/dom-to-semantic-markdown/blob/main/src/types/markdownTypes.ts"
  click Rollup "https://github.com/romansky/dom-to-semantic-markdown/blob/main/rollup.config.js"
  click TSBrowser "https://github.com/romansky/dom-to-semantic-markdown/blob/main/tsconfig.browser.json"
  click TSNode "https://github.com/romansky/dom-to-semantic-markdown/blob/main/tsconfig.node.json"
  click CI "https://github.com/romansky/dom-to-semantic-markdown/blob/main/.github/workflows/ci.yml"
  click Publish "https://github.com/romansky/dom-to-semantic-markdown/blob/main/.github/workflows/publish.yml"
  click Tests "https://github.com/romansky/dom-to-semantic-markdown/tree/main/tests/"
  click CLIExample "https://github.com/romansky/dom-to-semantic-markdown/tree/main/examples/cli"
  click BrowserExample "https://github.com/romansky/dom-to-semantic-markdown/blob/main/examples/browser.html"
  click NodeExample "https://github.com/romansky/dom-to-semantic-markdown/tree/main/examples/node"
```
