How snehify/substacklens works
SubstackLens indexes public RSS/Atom articles, then lets readers search them by keyword or semantic similarity, filter by publication, and page through results. Feed registration, manual and scheduled refresh, and ingestion status form a parallel management workflow. SQLite stores feeds and articles, with FTS5 maintained by triggers; semantic ranking embeds article metadata and queries. The frontend and backend excerpts support the search flow, but feed management is documented primarily as API capability, not as a frontend screen.
Main entry points: db.py SQLite store · main.py FastAPI endpoints · page.tsx Search interface · ingest.py Feed ingestion
Components 14
Reader discovery
- Search interface
frontend/app/page.tsx - Article results
frontend/app/page.tsx
API and orchestration
- FastAPI endpoints
backend/app/main.py - Feed management
backend/app/main.py - Refresh scheduler
backend/app/main.py
Feed indexing
- Feed ingestion
backend/app/ingest.py - SQLite store
backend/app/db.py
Search engines
- FTS5 index
backend/app/db.py - Keyword search
backend/app/main.py - Semantic search
backend/app/main.py - Embedding ranker
backend/app/semantic.py
Other components
- Reader
- RSS/Atom feeds
- Sentence embedding model
Connections 19
- Reader → Search interface: enters query
README.md - Search interface → FastAPI endpoints: requests search
frontend/app/page.tsx - FastAPI endpoints → Search interface: returns results
frontend/app/page.tsx - Search interface → Article results: renders results
frontend/app/page.tsx - FastAPI endpoints → SQLite store: reads and writes
backend/app/main.py - FastAPI endpoints → Keyword search: routes query
backend/app/main.py - Keyword search → FTS5 index: queries index
backend/app/main.py - FTS5 index → SQLite store: indexes article fields
backend/app/db.py - FastAPI endpoints → Semantic search: routes query
backend/app/main.py - Semantic search → Embedding ranker: ranks articles
backend/app/main.py - Semantic search → SQLite store: loads articles
backend/app/main.py - Embedding ranker → Sentence embedding model: encodes text
backend/app/semantic.py - Feed management → SQLite store: registers feeds
backend/app/main.py - Feed management → Feed ingestion: refreshes feeds
backend/app/main.py - Refresh scheduler → SQLite store: loads feeds
backend/app/main.py - Refresh scheduler → Feed ingestion: refreshes feeds
backend/app/main.py - Feed ingestion → RSS/Atom feeds: fetches feed data
backend/app/ingest.py - Feed ingestion → SQLite store: upserts articles
backend/app/ingest.py - Feed management → SQLite store: reads feed status
backend/app/main.py