How jgm/pandoc works
1. Repository shape
This is a multi-package Haskell project centered on the pandoc library. The top-level src/Text/Pandoc tree contains the core document model and transformation pipeline, while sibling packages (pandoc-cli, pandoc-server, pandoc-lua-engine) add delivery surfaces and scripting support. Supporting assets live under data/, and the repo is built/tested with Cabal, Stack, Nix, and CI workflows.
2. Core abstraction: the Pandoc AST
The architectural center is the native document AST defined in Text.Pandoc and related modules. Everything else is organized around converting into and out of this intermediate representation. That boundary is important: readers produce AST, transforms mutate AST, and writers consume AST.
3. Reader subsystem
src/Text/Pandoc/Readers/* contains format-specific parsers for many inputs: Markdown variants, HTML, LaTeX, Docx, EPUB, ODT, Org, Typst, XML-based formats, spreadsheets, and bibliography formats. Readers handle format quirks, normalize structure, and populate document metadata and embedded media references as they build the AST.
4. Writer subsystem
src/Text/Pandoc/Writers/* performs the inverse translation from AST to output formats such as HTML, LaTeX, DOCX, EPUB, PPTX, RTF, Org, Markdown variants, and many more. Writers are format-adapter boundaries: they decide how structural elements, tables, citations, notes, images, and templates map to the target syntax or container format.
5. Shared document processing pipeline
Between readers and writers sit shared modules for options, extensions, templates, slides, self-contained output, transforms, and source tracking. This is where pandoc applies document-level behavior such as formatting extensions, metadata handling, smart typography, sectioning, and output-specific rewriting before serialization.
6. Command-line frontend
pandoc-cli/src/pandoc.hs and the Text.Pandoc.App modules implement the main executable. This layer parses command-line options, resolves input/output sources, configures readers and writers, orchestrates the conversion pipeline, and handles standard I/O, files, defaults, filters, and exit behavior. It is the primary user-facing boundary for the library.
7. Lua scripting and custom extensions
pandoc-lua-engine is a separate package that embeds Lua for filters, custom readers/writers, and helper modules exposed to scripts. This layer bridges Haskell AST objects to Lua runtime values and back, enabling user-defined document processing without changing the core binary. It is also the mechanism behind the “custom Lua reader/writer” capability described in the README.
8. Bibliography and citation processing
The src/Text/Pandoc/Citeproc modules plus citeproc/ provide citation parsing, CSL data handling, name formatting, locators, and bibliography rendering. This subsystem sits logically between parsing and final writing: readers can preserve citation markup, then citeproc resolves and formats references before the writer emits the final document.
9. Embedded resources and format templates
The data/ directory is part of the runtime architecture, not just documentation. It contains default templates for many writers, translations/localization YAML files, CSL styles, DOCX/ODT/PPTX reference skeletons, completion scripts, and XML/schema helpers. Writers and the CLI load these assets to produce consistent output and localized messages.
10. I/O, media, and external resource boundaries
Text.Pandoc.Class, IO, Sandbox, MediaBag, Image, URI, and HTTP define how pandoc accesses files, remote resources, and embedded media while keeping pure and impure execution separated. This boundary matters for reproducibility and security: the core AST logic is distinct from fetching images, reading metadata files, or sandboxed evaluation.
11. Server and WebAssembly delivery surfaces
pandoc-server provides a server-oriented executable, while wasm/ packages pandoc for browser/WebAssembly use with example conversions and a JS wrapper. These are alternative frontends to the same core library and readers/writers, but they impose different runtime constraints around filesystem access, networking, and available binaries.
12. Platform packaging
The linux/, macos/, and windows/ trees contain packaging and installer logic, showing that pandoc is distributed as native binaries across platforms rather than as a web service. The packaging layer is downstream of the core conversion engine and mainly concerns distribution format, not document semantics.
13. Testing and regression coverage
test/ and pandoc-lua-engine/test/ contain extensive golden and command-based regression tests for readers, writers, CLI behavior, and round-trip edge cases. The large fixture set is a key architectural signal: correctness is enforced format-by-format, not just through unit tests, because compatibility across many document syntaxes is the product’s core risk.
14. How to draw the main flow
For an architecture diagram, the central path should be: input source → reader/parser → pandoc AST → optional citeproc/filters/transforms/Lua → writer/template engine → output target. Surrounding that path are supporting boundaries for options/configuration, embedded resources, media handling, and alternate frontends (CLI, server, WASM). The system is best understood as a conversion pipeline with many pluggable format adapters around one shared document model.
Main entry points: App.hs App modules · Pandoc.hs Pandoc AST · Transforms.hs Pipeline · Writers.hs Writers
Components 19
Core library
AST, parsing, writing, transforms, and shared runtime support.
- Pandoc AST · Native document representation at the center of the library.
src/Text/Pandoc.hs - Pipeline · Document options, extensions, templates, slides, self-contained rewriting, and source tracking.
src/Text/Pandoc/Transforms.hs - Readers · Format-specific parsers that normalize many input syntaxes into the AST.
src/Text/Pandoc/Readers.hs - Writers · Format-specific serializers that map the AST to target syntaxes and containers.
src/Text/Pandoc/Writers.hs - Citeproc · Citation parsing, CSL data handling, name formatting, and bibliography rendering.
src/Text/Pandoc/Citeproc.hs - I/O boundary · File, URI, HTTP, sandbox, and media handling outside the pure AST core.
src/Text/Pandoc/Class.hs - Lua bridge · Embedding and marshaling between Haskell document objects and Lua filters/readers/writers.
src/Text/Pandoc/Scripting.hs
Delivery surfaces
Executables and alternate runtimes built on the core library.
- CLI app · Main command-line executable and option-driven conversion orchestration.
pandoc-cli/src/pandoc.hs - App modules · Command-line parsing, input/output setup, defaults, filters, and exit behavior.
src/Text/Pandoc/App.hs - Server · Server-oriented executable exposing pandoc conversion as a service.
pandoc-server/src/Text/Pandoc/Server.hs - Lua engine · Standalone package providing Lua execution, modules, marshaling, and custom reader/writer support.
pandoc-lua-engine/src/Text/Pandoc/Lua.hs - WASM build · Browser/WebAssembly packaging and examples for running pandoc in constrained environments.
wasm/index.js
Runtime assets
Templates, translations, reference files, and bundled formats.
- Data assets · Bundled templates, translations, styles, skeleton documents, and other runtime resources.
data/templates/default.html5 - Citation data · CSL style, localization, and bibliography support data consumed by citeproc.
citeproc/biblatex-localization/english.lbx.strings
Quality and tests
Regression coverage and fixtures for readers, writers, and CLI behavior.
- Regression tests · Command-based and format-specific tests that validate readers, writers, and CLI behavior.
test/test-pandoc.hs - Lua tests · Lua-engine tests and script fixtures covering module APIs and custom behavior.
pandoc-lua-engine/test/test-pandoc-lua-engine.hs
Packaging
Platform-specific build and distribution tooling.
- Linux packaging · Native packaging and release artifact tooling for Linux builds.
linux/make_artifacts.sh - macOS packaging · Release and installer tooling for macOS builds.
macos/make_macos_release.sh - Windows packaging · WiX-based installer definitions and Windows release packaging.
windows/pandoc.wxs
Connections 25
- CLI app → App modules: uses · The executable delegates option parsing and orchestration to app modules.
- App modules → I/O boundary: reads/writes · CLI input and output are mediated through the I/O boundary.
- App modules → Readers: selects · Reader choice is configured from command-line options and defaults.
- App modules → Writers: selects · Writer choice is configured from command-line options and defaults.
- App modules → Pipeline: configures · The app layer assembles document-level processing behavior.
- Readers → Pandoc AST: produces · Readers parse source formats into the native AST.
- Pandoc AST → Pipeline: transforms · The AST is rewritten by shared document-processing stages.
- Pipeline → Citeproc: invokes · Citation processing runs as part of the document pipeline when enabled.
- Pipeline → Lua bridge: invokes · Filters and Lua extensions participate in the transformation pipeline.
- Pipeline → Data assets: loads · Templates, translations, and bundled resources influence rendering behavior.
- Writers → Data assets: uses · Writers load templates, reference skeletons, and localized assets.
- Writers → Pandoc AST: consumes · Writers serialize the AST into target formats.
- Citeproc → Citation data: uses · Citation formatting depends on CSL and localization data.
- I/O boundary → Data assets: resolves · Runtime resources and embedded media are accessed through this boundary.
- Server → Pandoc AST: uses · The server frontend exposes the same conversion core.
- Lua engine → Lua bridge: implements · The separate Lua package provides the execution and marshaling machinery.
- Lua engine → Pandoc AST: bridges · Lua modules operate on pandoc document values via the AST boundary.
- WASM build → Pandoc AST: uses · The WASM runtime reuses the same core library and format adapters.
- Regression tests → Readers: covers · Regression tests validate parser behavior across many formats.
- Regression tests → Writers: covers · Regression tests validate serializer output across many formats.
- Regression tests → App modules: covers · Command tests exercise CLI behavior and pipeline wiring.
- Lua tests → Lua engine: covers · Package tests exercise Lua modules, filters, and custom readers/writers.
- Linux packaging → CLI app: packages · Linux packaging wraps the CLI binary for distribution.
- macOS packaging → CLI app: packages · macOS packaging wraps the CLI binary for distribution.
- Windows packaging → CLI app: packages · Windows installers package the CLI binary for distribution.