How eugeneyan/obsidian-copilot works
Project type and purpose
• Obsidian-copilot is a full-stack retrieval-augmented generation (RAG) tool that integrates with Obsidian (a markdown note-taking app).
• It consists of two primary parts:
a. A Python-based backend service for indexing and retrieval (using OpenSearch and semantic search via Hugging Face embeddings).
b. An Obsidian plugin (TypeScript/JavaScript) that calls the backend to draft or reflect on notes in real time.
File structure
• Top-level:
• Dockerfile, docker-compose.yml, Makefile, build.sh – container build & orchestration
• src/ – Python backend code (FastAPI app, logger, and indexing scripts under src/prep)
• plugin/ – Obsidian plugin code (main.ts/js, manifest.json, styles.css, version bump, package.json)
• data/ – placeholder for any data artifacts
• assets/ – readme images
• requirements.txt – Python dependencies
• The presence of build_opensearch_index.py and build_semantic_index.py indicates two parallel indexing pipelines.
• docker-compose.yml spins up an OpenSearch container that the backend will use.
README for additional insights
• It describes RAG flow: build indices → start OpenSearch → run retrieval app → install plugin
• It references the Hugging Face cache and vault path environment variables
• Makefile targets: build (docker), opensearch (stand up container), build-artifacts (run indexing), run (start API), install-plugin
Main components & interactions
a. Obsidian Vault (user’s markdown notes)
b. Indexing Pipelines (Python scripts):
– build_opensearch_index.py → keyword-based indexing into OpenSearch
– build_semantic_index.py → embeddings-based vector store (could be a simple local store or wrapped by same service)
c. OpenSearch Container (launched via docker-compose)
d. Semantic Store / Hugging Face Cache (local filesystem)
e. Retrieval API Service (src/app.py, served by Uvicorn/FastAPI in a Docker container)
f. Obsidian Plugin (runs in Obsidian, written in TS/JS)
g. External Services (Hugging Face transformers, optionally the Internet for downloading models)
Relationships:
• Indexing Pipelines read from Obsidian Vault → push to OpenSearch & Semantic Store
• Retrieval API reads from both indices, merges results, and exposes REST endpoints
• Obsidian Plugin calls the Retrieval API over HTTP → renders suggestions in the editor
Architectural patterns & design principles
• Separation of concerns: indexing vs. retrieval vs. presentation (plugin)
• RAG pattern: blending sparse (keyword) and dense (embedding) search
• Containerization: each major service runs in its own container (API, OpenSearch)
• Extensibility: plugin manifest, version bump script, ESBuild config
Elements
• Labeled nodes: Obsidian Vault, Index Builder, OpenSearch, Semantic Store, Retrieval API, Obsidian Plugin
• Annotation callouts: “Built by build_opensearch_index.py”, “Built by build_semantic_index.py”, “Served via Uvicorn”