How ramxx/zettelvault works
1. What this repository is
ZettelVault is a Python batch-processing tool that transforms one or more Obsidian source vaults into a new destination vault organized by PARA and populated with atomic Zettelkasten notes. The core architecture is a linear pipeline with persistence checkpoints between stages so long runs can resume safely.
2. Top-level control plane
zettelvault/__main__.py provides the CLI entry point, and pipeline.py orchestrates the end-to-end flow: load config, initialize LLMs, read notes, classify, decompose, write files, then resolve dangling links. The CLI supports dry-run, sampling, skipping cached phases, and limiting the number of notes processed.
3. Vault ingestion boundary
vault_io.py is the adapter around the external vlt CLI, which is responsible for reading Obsidian vault contents and metadata from disk. This is an important boundary: ZettelVault does not parse Obsidian internals itself; it consumes structured note data from vlt, which also handles vault-name-to-path resolution and frontmatter/wikilink extraction.
4. Content normalization layer
sanitize.py prepares note text for model consumption by preserving frontmatter separately and escaping wikilinks so DSPy template syntax does not collide with Obsidian link syntax. This module is the main preservation boundary for knowledge graph fidelity: metadata and links are protected before any LLM call happens.
5. Classification subsystem
classify.py assigns each source note to PARA, plus a domain, subdomain, and tags. Architecturally, this phase is independent per note but intentionally runs before decomposition because the resulting organizational metadata and the concept index become inputs to later stages. Classification results are cached in classified_notes.json for resumeability.
6. Decomposition subsystem
decompose.py is the core intelligence layer. It turns one classified note into multiple atomic notes using DSPy, primarily dspy.RLM with a fallback chain to dspy.Predict and finally a passthrough single-note mode. This module also parses the model’s markdown-delimited atom output, validates it, and records fallback events in fallback_notes.json.
7. RLM execution boundary
The RLM path is architecturally distinct from normal prompting: the note content is placed into a sandboxed REPL environment backed by Deno/Pyodide, and the model writes Python code to inspect and split the document programmatically. That makes the decomposition step more like an analysis engine than a text generator, and it is the reason the project can handle long, structured notes with reliable atomic splitting.
8. Write path and vault materialization
writer.py converts atomic note objects into actual Markdown files in the destination vault. It preserves original frontmatter fields where possible, adds generated metadata, creates PARA folder paths, handles filename collisions safely, and generates one Map-of-Content note per domain. This is the main persistence boundary where in-memory note objects become the new vault structure.
9. Link cleanup and graph repair
resolve.py runs after writing to scan the destination vault for orphan wikilinks. It repairs the graph using case-insensitive matching, fuzzy matching, stub-note creation for heavily referenced missing targets, or link removal when references are sparse. This step is what turns a partially generated vault into a navigable Obsidian graph.
10. Configuration and runtime wiring
config.py loads config.yaml plus optional local overrides and feeds model/runtime settings into the pipeline. The config controls model provider details, RLM limits, retry behavior, checkpoint cadence, sampling thresholds, and link-resolution rules. This makes the pipeline adaptable to different LLM backends, including OpenRouter and other OpenAI-compatible endpoints.
11. Cost and usage telemetry
pricing.py is a separate observability module that reads DSPy history and pairs token usage with model pricing fetched from OpenRouter. It does not affect the transformation itself, but it is architecturally important because the system is designed for large batch runs where phase-level cost accounting and provider-reported totals matter.
12. Sampling and operational safety
sample.py supports generating a representative subset of notes for previewing the pipeline without processing the full vault. Across the repo, safety mechanisms include checkpoint files, dry-run mode, fallback logging, filename sanitization, and non-destructive config layering. Together these make the system resilient for multi-hour migrations.
13. External dependencies that shape the architecture
The main external services and tools are Obsidian-compatible vault storage on disk, the vlt vault CLI for structured reading/writing, DSPy for LLM orchestration, a model provider such as OpenRouter or an OpenAI-compatible endpoint, and Deno for the RLM sandbox. These are not incidental dependencies; they define the major system boundaries.
14. Testing structure
The tests/ directory focuses on the core transformation boundaries: sanitization, parsing, filename safety, writing, vault I/O, and cost tracking. That test layout mirrors the architecture, which is centered on deterministic preprocessing/postprocessing around nondeterministic LLM calls.
Main entry points: pipeline.py Pipeline · vault_io.py Vault I/O · decompose.py Decompose · classify.py Classify
Components 16
Control plane
CLI entrypoint and end-to-end orchestration
Ingest & normalize
Read source vault data and protect note structure before LLM calls
Transform core
Per-note classification and decomposition into atomic notes
Write & repair
Materialize destination vault and clean up links
Support
Configuration, sampling, telemetry, and tests
External systems
Tools and providers that define major boundaries
- DSPy · Orchestrates prompting and RLM/predict fallback execution
- RLM sandbox · Sandboxed REPL environment used for programmatic splitting
- vlt CLI · External vault reader/writer and metadata extractor
- Model API · OpenRouter or OpenAI-compatible endpoint
- Vault files · Source and destination Obsidian vaults on disk
Connections 19
- CLI → Pipeline: starts · CLI hands execution to the orchestrator
- Pipeline → Config: loads · Reads runtime settings and overrides
- Pipeline → Sampling: optionally samples · Can reduce the source set for preview runs
- Pipeline → Vault I/O: reads · Pulls structured notes from source vaults
- Vault I/O → vlt CLI: uses · Delegates disk vault access to the external CLI
- vlt CLI → Vault files: reads · Consumes source vault contents and metadata
- Vault I/O → Sanitize: normalizes · Prepares note text for model input
- Pipeline → Classify: classifies · Assigns organization metadata before decomposition
- Classify → DSPy: calls · Uses DSPy for model-backed classification
- Pipeline → Decompose: decomposes · Splits classified notes into atomic notes
- Decompose → DSPy: orchestrates · Runs the main LLM-driven decomposition flow
- Decompose → RLM sandbox: executes in · Primary path uses sandboxed program analysis
- RLM sandbox → Model API: queries · Model generates code or analysis decisions
- DSPy → Model API: queries · Fallback predict path uses the model backend
- Pipeline → Writer: writes · Materializes atomic notes into destination vault files
- Writer → Vault files: creates · Produces the new vault structure on disk
- Pipeline → Resolve: repairs links · Runs graph cleanup after writing
- Resolve → Vault files: updates · Scans and edits destination vault files
- Pipeline → Pricing: reports cost · Can summarize token usage and spend after runs