# ramxx/zettelvault architecture

> Architecture diagram of the GitHub repository ramxx/zettelvault, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ramxx/zettelvault
- Repository: https://github.com/ramxx/zettelvault

## Overview

1. **What this repository is**
   ZettelVault is a Python batch-processing tool that transforms one or more Obsidian source vaults into a new destination vault organized by PARA and populated with atomic Zettelkasten notes. The core architecture is a linear pipeline with persistence checkpoints between stages so long runs can resume safely.

2. **Top-level control plane**
   `zettelvault/__main__.py` provides the CLI entry point, and `pipeline.py` orchestrates the end-to-end flow: load config, initialize LLMs, read notes, classify, decompose, write files, then resolve dangling links. The CLI supports dry-run, sampling, skipping cached phases, and limiting the number of notes processed.

3. **Vault ingestion boundary**
   `vault_io.py` is the adapter around the external `vlt` CLI, which is responsible for reading Obsidian vault contents and metadata from disk. This is an important boundary: ZettelVault does not parse Obsidian internals itself; it consumes structured note data from vlt, which also handles vault-name-to-path resolution and frontmatter/wikilink extraction.

4. **Content normalization layer**
   `sanitize.py` prepares note text for model consumption by preserving frontmatter separately and escaping wikilinks so DSPy template syntax does not collide with Obsidian link syntax. This module is the main preservation boundary for knowledge graph fidelity: metadata and links are protected before any LLM call happens.

5. **Classification subsystem**
   `classify.py` assigns each source note to PARA, plus a domain, subdomain, and tags. Architecturally, this phase is independent per note but intentionally runs before decomposition because the resulting organizational metadata and the concept index become inputs to later stages. Classification results are cached in `classified_notes.json` for resumeability.

6. **Decomposition subsystem**
   `decompose.py` is the core intelligence layer. It turns one classified note into multiple atomic notes using DSPy, primarily `dspy.RLM` with a fallback chain to `dspy.Predict` and finally a passthrough single-note mode. This module also parses the model’s markdown-delimited atom output, validates it, and records fallback events in `fallback_notes.json`.

7. **RLM execution boundary**
   The RLM path is architecturally distinct from normal prompting: the note content is placed into a sandboxed REPL environment backed by Deno/Pyodide, and the model writes Python code to inspect and split the document programmatically. That makes the decomposition step more like an analysis engine than a text generator, and it is the reason the project can handle long, structured notes with reliable atomic splitting.

8. **Write path and vault materialization**
   `writer.py` converts atomic note objects into actual Markdown files in the destination vault. It preserves original frontmatter fields where possible, adds generated metadata, creates PARA folder paths, handles filename collisions safely, and generates one Map-of-Content note per domain. This is the main persistence boundary where in-memory note objects become the new vault structure.

9. **Link cleanup and graph repair**
   `resolve.py` runs after writing to scan the destination vault for orphan wikilinks. It repairs the graph using case-insensitive matching, fuzzy matching, stub-note creation for heavily referenced missing targets, or link removal when references are sparse. This step is what turns a partially generated vault into a navigable Obsidian graph.

10. **Configuration and runtime wiring**
   `config.py` loads `config.yaml` plus optional local overrides and feeds model/runtime settings into the pipeline. The config controls model provider details, RLM limits, retry behavior, checkpoint cadence, sampling thresholds, and link-resolution rules. This makes the pipeline adaptable to different LLM backends, including OpenRouter and other OpenAI-compatible endpoints.

11. **Cost and usage telemetry**
   `pricing.py` is a separate observability module that reads DSPy history and pairs token usage with model pricing fetched from OpenRouter. It does not affect the transformation itself, but it is architecturally important because the system is designed for large batch runs where phase-level cost accounting and provider-reported totals matter.

12. **Sampling and operational safety**
   `sample.py` supports generating a representative subset of notes for previewing the pipeline without processing the full vault. Across the repo, safety mechanisms include checkpoint files, dry-run mode, fallback logging, filename sanitization, and non-destructive config layering. Together these make the system resilient for multi-hour migrations.

13. **External dependencies that shape the architecture**
   The main external services and tools are Obsidian-compatible vault storage on disk, the `vlt` vault CLI for structured reading/writing, DSPy for LLM orchestration, a model provider such as OpenRouter or an OpenAI-compatible endpoint, and Deno for the RLM sandbox. These are not incidental dependencies; they define the major system boundaries.

14. **Testing structure**
   The `tests/` directory focuses on the core transformation boundaries: sanitization, parsing, filename safety, writing, vault I/O, and cost tracking. That test layout mirrors the architecture, which is centered on deterministic preprocessing/postprocessing around nondeterministic LLM calls.

Main entry points: [`zettelvault/pipeline.py`](https://github.com/ramxx/zettelvault/blob/main/zettelvault/pipeline.py) (Pipeline), [`zettelvault/vault_io.py`](https://github.com/ramxx/zettelvault/blob/main/zettelvault/vault_io.py) (Vault I/O), [`zettelvault/decompose.py`](https://github.com/ramxx/zettelvault/blob/main/zettelvault/decompose.py) (Decompose), [`zettelvault/classify.py`](https://github.com/ramxx/zettelvault/blob/main/zettelvault/classify.py) (Classify)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_control["Control plane"]
  node_n_main(("CLI<br/>entrypoint<br/>[__main__.py]"))
  node_n_pipeline["Pipeline<br/>orchestrator<br/>[pipeline.py]"]
end

subgraph group_grp_ingest["Ingest & normalize"]
  node_n_vaultio[("Vault I/O<br/>vault adapter<br/>[vault_io.py]")]
  node_n_sanitize["Sanitize<br/>text guard<br/>[sanitize.py]"]
end

subgraph group_grp_intel["Transform core"]
  node_n_classify["Classify<br/>note classifier<br/>[classify.py]"]
  node_n_decompose["Decompose<br/>atomizer<br/>[decompose.py]"]
end

subgraph group_grp_write["Write & repair"]
  node_n_resolve["Resolve<br/>link repair<br/>[resolve.py]"]
  node_n_writer[("Writer<br/>vault builder<br/>[writer.py]")]
end

subgraph group_grp_support["Support"]
  node_n_config["Config<br/>[config.py]"]
  node_n_sample["Sampling<br/>preview helper<br/>[sample.py]"]
  node_n_pricing["Pricing<br/>telemetry<br/>[pricing.py]"]
end

subgraph group_grp_external["External systems"]
  node_n_dspy{{"DSPy<br/>llm framework"}}
  node_n_rlm{{"RLM sandbox<br/>analysis runtime"}}
  node_n_vlt["vlt CLI<br/>vault tool"]
  node_n_provider[("Model API<br/>llm provider")]
  node_n_obsidian[("Vault files<br/>storage")]
end

node_n_main -->|"starts"| node_n_pipeline
node_n_pipeline -->|"loads"| node_n_config
node_n_pipeline -->|"optionally samples"| node_n_sample
node_n_pipeline -->|"reads"| node_n_vaultio
node_n_vaultio -->|"uses"| node_n_vlt
node_n_vlt -->|"reads"| node_n_obsidian
node_n_vaultio -->|"normalizes"| node_n_sanitize
node_n_pipeline -->|"classifies"| node_n_classify
node_n_classify -->|"calls"| node_n_dspy
node_n_pipeline -->|"decomposes"| node_n_decompose
node_n_decompose -->|"orchestrates"| node_n_dspy
node_n_decompose -->|"executes in"| node_n_rlm
node_n_rlm -->|"queries"| node_n_provider
node_n_dspy -->|"queries"| node_n_provider
node_n_pipeline -->|"writes"| node_n_writer
node_n_writer -->|"creates"| node_n_obsidian
node_n_pipeline -->|"repairs links"| node_n_resolve
node_n_resolve -->|"updates"| node_n_obsidian
node_n_pipeline -->|"reports cost"| node_n_pricing

click node_n_main "https://github.com/ramxx/zettelvault/blob/main/zettelvault/__main__.py"
click node_n_pipeline "https://github.com/ramxx/zettelvault/blob/main/zettelvault/pipeline.py"
click node_n_config "https://github.com/ramxx/zettelvault/blob/main/zettelvault/config.py"
click node_n_sample "https://github.com/ramxx/zettelvault/blob/main/zettelvault/sample.py"
click node_n_vaultio "https://github.com/ramxx/zettelvault/blob/main/zettelvault/vault_io.py"
click node_n_sanitize "https://github.com/ramxx/zettelvault/blob/main/zettelvault/sanitize.py"
click node_n_classify "https://github.com/ramxx/zettelvault/blob/main/zettelvault/classify.py"
click node_n_decompose "https://github.com/ramxx/zettelvault/blob/main/zettelvault/decompose.py"
click node_n_resolve "https://github.com/ramxx/zettelvault/blob/main/zettelvault/resolve.py"
click node_n_writer "https://github.com/ramxx/zettelvault/blob/main/zettelvault/writer.py"
click node_n_pricing "https://github.com/ramxx/zettelvault/blob/main/pricing.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n_main,node_n_pipeline toneBlue
class node_n_vaultio,node_n_sanitize toneAmber
class node_n_classify,node_n_decompose toneMint
class node_n_resolve,node_n_writer toneRose
class node_n_config,node_n_sample,node_n_pricing toneIndigo
class node_n_dspy,node_n_rlm,node_n_vlt,node_n_provider,node_n_obsidian toneTeal
```

## Components

### Control plane

CLI entrypoint and end-to-end orchestration

- [CLI](https://github.com/ramxx/zettelvault/blob/main/zettelvault/__main__.py): entrypoint · Command-line entry for running the pipeline · `zettelvault/__main__.py`
- [Pipeline](https://github.com/ramxx/zettelvault/blob/main/zettelvault/pipeline.py): orchestrator · Coordinates load, process, write, and repair stages · `zettelvault/pipeline.py`

### Ingest & normalize

Read source vault data and protect note structure before LLM calls

- [Vault I/O](https://github.com/ramxx/zettelvault/blob/main/zettelvault/vault_io.py): vault adapter · Reads structured notes and metadata from source vaults · `zettelvault/vault_io.py`
- [Sanitize](https://github.com/ramxx/zettelvault/blob/main/zettelvault/sanitize.py): text guard · Preserves frontmatter and escapes links before model use · `zettelvault/sanitize.py`

### Transform core

Per-note classification and decomposition into atomic notes

- [Classify](https://github.com/ramxx/zettelvault/blob/main/zettelvault/classify.py): note classifier · Assigns PARA, domain, subdomain, and tags per source note · `zettelvault/classify.py`
- [Decompose](https://github.com/ramxx/zettelvault/blob/main/zettelvault/decompose.py): atomizer · Splits a classified note into atomic notes with fallback modes · `zettelvault/decompose.py`

### Write & repair

Materialize destination vault and clean up links

- [Resolve](https://github.com/ramxx/zettelvault/blob/main/zettelvault/resolve.py): link repair · Repairs or trims dangling links in the destination vault · `zettelvault/resolve.py`
- [Writer](https://github.com/ramxx/zettelvault/blob/main/zettelvault/writer.py): vault builder · Creates Markdown notes, folder paths, and MOCs · `zettelvault/writer.py`

### Support

Configuration, sampling, telemetry, and tests

- [Config](https://github.com/ramxx/zettelvault/blob/main/zettelvault/config.py): runtime config · Loads YAML config and runtime overrides · `zettelvault/config.py`
- [Sampling](https://github.com/ramxx/zettelvault/blob/main/zettelvault/sample.py): preview helper · Selects representative notes for smaller runs · `zettelvault/sample.py`
- [Pricing](https://github.com/ramxx/zettelvault/blob/main/pricing.py): telemetry · Combines DSPy usage with model pricing for cost accounting · `pricing.py`

### External systems

Tools and providers that define major boundaries

- **DSPy**: llm framework · Orchestrates prompting and RLM/predict fallback execution
- **RLM sandbox**: analysis runtime · Sandboxed REPL environment used for programmatic splitting
- **vlt CLI**: vault tool · External vault reader/writer and metadata extractor
- **Model API**: llm provider · OpenRouter or OpenAI-compatible endpoint
- **Vault files**: storage · Source and destination Obsidian vaults on disk

## Connections

- CLI → Pipeline: starts · CLI hands execution to the orchestrator
- Pipeline → Config: loads · Reads runtime settings and overrides
- Pipeline → Sampling: optionally samples · Can reduce the source set for preview runs
- Pipeline → Vault I/O: reads · Pulls structured notes from source vaults
- Vault I/O → vlt CLI: uses · Delegates disk vault access to the external CLI
- vlt CLI → Vault files: reads · Consumes source vault contents and metadata
- Vault I/O → Sanitize: normalizes · Prepares note text for model input
- Pipeline → Classify: classifies · Assigns organization metadata before decomposition
- Classify → DSPy: calls · Uses DSPy for model-backed classification
- Pipeline → Decompose: decomposes · Splits classified notes into atomic notes
- Decompose → DSPy: orchestrates · Runs the main LLM-driven decomposition flow
- Decompose → RLM sandbox: executes in · Primary path uses sandboxed program analysis
- RLM sandbox → Model API: queries · Model generates code or analysis decisions
- DSPy → Model API: queries · Fallback predict path uses the model backend
- Pipeline → Writer: writes · Materializes atomic notes into destination vault files
- Writer → Vault files: creates · Produces the new vault structure on disk
- Pipeline → Resolve: repairs links · Runs graph cleanup after writing
- Resolve → Vault files: updates · Scans and edits destination vault files
- Pipeline → Pricing: reports cost · Can summarize token usage and spend after runs
