# snehify/substacklens architecture

> Architecture diagram of the GitHub repository snehify/substacklens, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/snehify/substacklens
- Repository: https://github.com/snehify/substacklens

## Overview

SubstackLens indexes public RSS/Atom articles, then lets readers search them by keyword or semantic similarity, filter by publication, and page through results. Feed registration, manual and scheduled refresh, and ingestion status form a parallel management workflow. SQLite stores feeds and articles, with FTS5 maintained by triggers; semantic ranking embeds article metadata and queries. The frontend and backend excerpts support the search flow, but feed management is documented primarily as API capability, not as a frontend screen.

Main entry points: [`backend/app/db.py`](https://github.com/snehify/substacklens/blob/main/backend/app/db.py) (SQLite store), [`backend/app/main.py`](https://github.com/snehify/substacklens/blob/main/backend/app/main.py) (FastAPI endpoints), [`frontend/app/page.tsx`](https://github.com/snehify/substacklens/blob/main/frontend/app/page.tsx) (Search interface), [`backend/app/ingest.py`](https://github.com/snehify/substacklens/blob/main/backend/app/ingest.py) (Feed ingestion)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_discovery["Reader discovery"]
  node_search_ui["Search interface<br/>[page.tsx]"]
  node_article_results["Article results<br/>[page.tsx]"]
end

subgraph group_api["API and orchestration"]
  node_api["FastAPI endpoints<br/>[main.py]"]
  node_feed_admin["Feed management<br/>[main.py]"]
  node_refresh_scheduler["Refresh scheduler<br/>[main.py]"]
end

subgraph group_indexing["Feed indexing"]
  node_feed_ingestion["Feed ingestion<br/>[ingest.py]"]
  node_sqlite[("SQLite store<br/>[db.py]")]
end

subgraph group_search["Search engines"]
  node_fts5[("FTS5 index<br/>[db.py]")]
  node_keyword_search["Keyword search<br/>[main.py]"]
  node_semantic_search["Semantic search<br/>[main.py]"]
  node_semantic_ranker["Embedding ranker<br/>[semantic.py]"]
end

node_reader(("Reader"))
node_public_feeds["RSS/Atom feeds"]
node_embedding_model["Sentence embedding model"]

node_reader -->|"enters query"| node_search_ui
node_search_ui -->|"requests search"| node_api
node_api -->|"returns results"| node_search_ui
node_search_ui -->|"renders results"| node_article_results
node_api -->|"reads and writes"| node_sqlite
node_api -->|"routes query"| node_keyword_search
node_keyword_search -->|"queries index"| node_fts5
node_fts5 -->|"indexes article fields"| node_sqlite
node_api -->|"routes query"| node_semantic_search
node_semantic_search -->|"ranks articles"| node_semantic_ranker
node_semantic_search -->|"loads articles"| node_sqlite
node_semantic_ranker -->|"encodes text"| node_embedding_model
node_feed_admin -->|"registers feeds"| node_sqlite
node_feed_admin -->|"refreshes feeds"| node_feed_ingestion
node_refresh_scheduler -->|"loads feeds"| node_sqlite
node_refresh_scheduler -->|"refreshes feeds"| node_feed_ingestion
node_feed_ingestion -->|"fetches feed data"| node_public_feeds
node_feed_ingestion -->|"upserts articles"| node_sqlite
node_feed_admin -->|"reads feed status"| node_sqlite

click node_search_ui "https://github.com/snehify/substacklens/blob/main/frontend/app/page.tsx"
click node_article_results "https://github.com/snehify/substacklens/blob/main/frontend/app/page.tsx"
click node_api "https://github.com/snehify/substacklens/blob/main/backend/app/main.py"
click node_feed_admin "https://github.com/snehify/substacklens/blob/main/backend/app/main.py"
click node_refresh_scheduler "https://github.com/snehify/substacklens/blob/main/backend/app/main.py"
click node_feed_ingestion "https://github.com/snehify/substacklens/blob/main/backend/app/ingest.py"
click node_sqlite "https://github.com/snehify/substacklens/blob/main/backend/app/db.py"
click node_fts5 "https://github.com/snehify/substacklens/blob/main/backend/app/db.py"
click node_keyword_search "https://github.com/snehify/substacklens/blob/main/backend/app/main.py"
click node_semantic_search "https://github.com/snehify/substacklens/blob/main/backend/app/main.py"
click node_semantic_ranker "https://github.com/snehify/substacklens/blob/main/backend/app/semantic.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_search_ui,node_article_results toneBlue
class node_api,node_feed_admin,node_refresh_scheduler toneAmber
class node_feed_ingestion,node_sqlite toneMint
class node_fts5,node_keyword_search,node_semantic_search,node_semantic_ranker toneRose
class node_reader,node_public_feeds,node_embedding_model toneIndigo
```

## Components

### Reader discovery

- [Search interface](https://github.com/snehify/substacklens/blob/main/frontend/app/page.tsx): `frontend/app/page.tsx`
- [Article results](https://github.com/snehify/substacklens/blob/main/frontend/app/page.tsx): `frontend/app/page.tsx`

### API and orchestration

- [FastAPI endpoints](https://github.com/snehify/substacklens/blob/main/backend/app/main.py): `backend/app/main.py`
- [Feed management](https://github.com/snehify/substacklens/blob/main/backend/app/main.py): `backend/app/main.py`
- [Refresh scheduler](https://github.com/snehify/substacklens/blob/main/backend/app/main.py): `backend/app/main.py`

### Feed indexing

- [Feed ingestion](https://github.com/snehify/substacklens/blob/main/backend/app/ingest.py): `backend/app/ingest.py`
- [SQLite store](https://github.com/snehify/substacklens/blob/main/backend/app/db.py): `backend/app/db.py`

### Search engines

- [FTS5 index](https://github.com/snehify/substacklens/blob/main/backend/app/db.py): `backend/app/db.py`
- [Keyword search](https://github.com/snehify/substacklens/blob/main/backend/app/main.py): `backend/app/main.py`
- [Semantic search](https://github.com/snehify/substacklens/blob/main/backend/app/main.py): `backend/app/main.py`
- [Embedding ranker](https://github.com/snehify/substacklens/blob/main/backend/app/semantic.py): `backend/app/semantic.py`

### Other components

- **Reader**
- **RSS/Atom feeds**
- **Sentence embedding model**

## Connections

- Reader → Search interface: enters query (evidence: `README.md`)
- Search interface → FastAPI endpoints: requests search (evidence: `frontend/app/page.tsx`)
- FastAPI endpoints → Search interface: returns results (evidence: `frontend/app/page.tsx`)
- Search interface → Article results: renders results (evidence: `frontend/app/page.tsx`)
- FastAPI endpoints → SQLite store: reads and writes (evidence: `backend/app/main.py`)
- FastAPI endpoints → Keyword search: routes query (evidence: `backend/app/main.py`)
- Keyword search → FTS5 index: queries index (evidence: `backend/app/main.py`)
- FTS5 index → SQLite store: indexes article fields (evidence: `backend/app/db.py`)
- FastAPI endpoints → Semantic search: routes query (evidence: `backend/app/main.py`)
- Semantic search → Embedding ranker: ranks articles (evidence: `backend/app/main.py`)
- Semantic search → SQLite store: loads articles (evidence: `backend/app/main.py`)
- Embedding ranker → Sentence embedding model: encodes text (evidence: `backend/app/semantic.py`)
- Feed management → SQLite store: registers feeds (evidence: `backend/app/main.py`)
- Feed management → Feed ingestion: refreshes feeds (evidence: `backend/app/main.py`)
- Refresh scheduler → SQLite store: loads feeds (evidence: `backend/app/main.py`)
- Refresh scheduler → Feed ingestion: refreshes feeds (evidence: `backend/app/main.py`)
- Feed ingestion → RSS/Atom feeds: fetches feed data (evidence: `backend/app/ingest.py`)
- Feed ingestion → SQLite store: upserts articles (evidence: `backend/app/ingest.py`)
- Feed management → SQLite store: reads feed status (evidence: `backend/app/main.py`)