How mwouts/jupytext works
Project type and purpose
• Jupytext is an open-source tool (Python library + JupyterLab extension + CLI) that • converts Jupyter notebooks (.ipynb) to/from plain-text “text notebooks” (e.g. .py, .md, etc.) • synchronizes paired files • integrates with Jupyter Server and JupyterLab for on-the-fly conversion.
File structure overview
• Top level
• src/jupytext: core conversion engine, CLI entry point, file-format readers/writers, content manager adapters
• src/jupytext_config: configuration loader for global or per-directory settings
• jupyterlab/...: a full JupyterLab extension (TypeScript + schema + UI commands)
• docs/, demo/: documentation site + demo notebooks
• tests/: unit, functional, integration tests for every component
• CI, GitHub workflows, binder: build, test, publish pipelines
• Patterns
• Core library separated from CLI (__main__.py, cli.py)
• Jupyter Server extension via sync/async content managers
• Frontend/ backend split in JupyterLab extension (TS code + server-side install scripts)
Key architecture components
a. Core Conversion Engine (Python)
• Modules: formats.py, cell_reader.py, cell_to_text.py, pairs.py, pandoc.py, etc.
• Responsibilities: parse input (ipynb, markdown, script), map cells ↔ text, emit output in target format.
b. Configuration Loader
• jupytext_config.py: reads jupytext.toml or environment settings, exposes format mappings.
c. CLI Layer
• cli.py & __main__.py: command parsing, invokes core engine for --to, --sync, --set-formats, --pipe.
d. Jupyter Server Extension
• sync_contentsmanager.py & async_contentsmanager.py: plug into Jupyter ContentsManager API to intercept file I/O, convert on save/load.
e. JupyterLab Frontend Extension
• Located under jupyterlab/packages/jupyterlab-jupytext-extension/src: UI commands, toolbar buttons, right-click “Pair Notebook” actions • communicates via Jupyter REST API to server extension.
f. File System (User notebooks)
• Source files (.ipynb, .py, .md, …) stored on disk or in remote storage.
g. External Integrations
• Pre-commit hooks, black/flake8 integrations (--pipe), binder launch, CI pipelines (GitHub Actions, codecov).
Interactions and data flow
• CLI flow: User → CLI wrapper → Core engine → Read input file(s) → Convert → Write output file
• JupyterLab flow:
1. User opens file in JupyterLab frontend
2. Frontend command issues HTTP request to Jupyter server extension
3. Server extension uses core engine to convert on-the-fly
4. Notebook is loaded as .ipynb in browser; saves trigger reverse conversion
• Configuration: all flows consult the configuration loader for format rules before invoking the engine.
Architectural patterns and principles
• Separation of concerns: conversion core, CLI, server-extension, frontend-extension each in its own module
• Plugin pattern: Jupyter ContentsManager API and JupyterLab extension schema allow pluggable integration
• Modular design: format handlers isolated, new formats added via formats.py registry
• Test-driven: extensive unit, functional and integration tests demonstrate each component’s contract.
Final checklist
• Components above with clear labels
• Conversion pipeline, configuration lookup, and extension points
Top Layer: User Interfaces (CLI; JupyterLab UI)
Middle Layer: Integration & Extensions (CLI wrapper; Server Extension)
Core Layer: Conversion Engine & Config Loader
Bottom Layer: Storage (Notebook files; Config files)
External: Pre-commit, CI pipelines to the side.