How romansky/dom-to-semantic-markdown works
Project Type and Purpose
• This is an open-source JavaScript/TypeScript library (dom-to-semantic-markdown) that converts HTML DOM into a semantic Markdown AST/string optimized for LLM consumption.
• It provides three primary “faces”:
• A core conversion engine (reusable API)
• A CLI wrapper for fetching HTML (via HTTP or Playwright), calling the core, and writing Markdown
• A browser-side API (for in-page conversion)
File Structure Analysis
• Top-Level Config & Metadata:
• .github/workflows → CI (test, lint) and publish pipelines
• package.json, rollup.config.js → bundling/browser+node builds, npm publish
• jest.config.js, tsconfig.*.json → test/build configs
• src/
• src/index.ts → public API exports and CLI entry
• src/core/ → core engine modules (AST utils, DOM traversal, HTML→AST, AST→string, URL utils)
• src/types/ → shared TypeScript types (SemanticMarkdownAST, ConversionOptions)
• tests/ → unit tests for core functionality
• examples/ → sample CLI, browser, and Node.js usage
• issue.p, failing-test.p → repro test cases
Main Architectural Components
A. CLI Layer
• Entry script (src/index.ts) invoked by bin/d2m.
• Parses flags (–url, –input, –extract-main, –track-table-columns, –include-meta-data, –use-playwright).
• If URL: either HTTP fetch or Playwright fetch → raw HTML.
• Calls Core API → returns Markdown string → writes to stdout/file.
B. Core Conversion Engine
• HTML parsing → DOM (browser DOMParser or jsdom)
• htmlToMarkdownAST (src/core/htmlToMarkdownAST.ts) → SemanticMarkdownAST[]
• markdownASTToString (src/core/markdownASTToString.ts) → Markdown text
• astUtils, domUtils, urlUtils aid transformations, metadata extraction, URL refification.
• Extensibility via overrideElementProcessing, processUnhandledElement, overrideNodeRenderer, renderCustomNode options.
C. Browser and Node API
• Browser: convertHtmlToMarkdown(document.body, options) → Markdown
• Node: convertHtmlToMarkdown(string HTML, options + overrideDOMParser via jsdom)
D. Build & Publish Pipeline
• Rollup builds: outputs CJS/ESM bundles for Node and browser (tsconfigs for each)
• GitHub Actions:
• CI workflow → runs jest tests, lint, typecheck
• Publish workflow → on npm release tags
Interactions and Data Flows
• CLI Input → (HTTP / Playwright) → HTML → Core Engine → Markdown → CLI Output
• Browser API: DOM Element → Core Engine → Markdown
• Node API: HTML String → jsdom DOM → Core Engine → Markdown
• Core Engine publishes AST → can be intercepted by custom renderers/plugins
Architectural Patterns & Principles
• Separation of Concerns: CLI, core logic, bundling, examples, and tests all live in separate folders
• Plugin-style overrides: dependency injection for element processing/rendering
• Build-time code splitting: separate browser and Node bundles via Rollup/tsconfig
• Test-driven: jest tests for each core feature (metadata extraction, main content detection, table tracking)
Technologies & Frameworks
• TypeScript, Rollup, Jest
• Node.js, ECMAScript modules, npm CLI
• jsdom (Node DOM parsing), optional Playwright (dynamic fetch)