# wl2bd/thothly architecture

> Architecture diagram of the GitHub repository wl2bd/thothly, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wl2bd/thothly
- Repository: https://github.com/wl2bd/thothly

## Overview

Thothly turns searched or pasted web, video, and podcast sources into a reviewed compilation, then delivers an EPUB and Markdown twin. The principal workflow is search/paste → discovery → review and selection → compilation → download/read. The graph preserves search, source extraction, optional AI polish and podcast transcription, rendering, job persistence, and the browser workspace. Some backend orchestration is supported by the SOURCE INDEX and README rather than inspected implementation; edges are limited to those references and documented relationships.

Main entry points: [`backend/app/jobs/runner.py`](https://github.com/wl2bd/thothly/blob/main/backend/app/jobs/runner.py) (Job runner), [`backend/app/jobs/router.py`](https://github.com/wl2bd/thothly/blob/main/backend/app/jobs/router.py) (Job routes), [`frontend/components/compose.tsx`](https://github.com/wl2bd/thothly/blob/main/frontend/components/compose.tsx) (Compose workspace), [`frontend/app/jobs/[id]/page.tsx`](https://github.com/wl2bd/thothly/blob/main/frontend/app/jobs/%5Bid%5D/page.tsx) (Compilation reader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Browser workspace"]
  node_workspace["Compose workspace<br/>[compose.tsx]"]
  node_job_page["Compilation reader<br/>[page.tsx]"]
  node_browser_history[("Browser history<br/>[history.ts]")]
end

subgraph group_search["Search and discovery"]
  node_search_service["Search aggregation<br/>[service.py]"]
  node_search_providers["Search providers"]
  node_triage["Search triage<br/>[triage.py]"]
  node_discovery["Source discovery<br/>[discovery.py]"]
end

subgraph group_jobs["Compilation jobs"]
  node_job_api["Job routes<br/>[router.py]"]
  node_preview["Review previews<br/>[preview.py]"]
  node_job_runner["Job runner<br/>[runner.py]"]
  node_job_repository["Job repository<br/>[repository.py]"]
  node_download["Book download<br/>[router.py]"]
end

subgraph group_content["Content processing"]
  node_source_extractors["Source extractors"]
  node_transcript_cache[("Transcript caches")]
  node_cleanup["Text cleanup<br/>[cleanup.py]"]
  node_llm["LLM client<br/>[llm.py]"]
  node_stt["Podcast transcription<br/>[transcribe.py]"]
  node_compiler["Content compiler<br/>[compiler.py]"]
  node_renderer["EPUB and cover renderer<br/>[epub.py]"]
end

subgraph group_platform["Persistence and API"]
  node_search_api["Search API<br/>[search.py]"]
  node_sqlite[("SQLite database<br/>[database.py]")]
  node_backend_app["Backend application<br/>[main.py]"]
end

node_reader(("User"))

node_reader -->|"searches or pastes"| node_workspace
node_workspace -->|"searches"| node_search_api
node_workspace -->|"starts job"| node_job_api
node_job_api -->|"creates and reads"| node_job_repository
node_job_api -->|"starts discovery"| node_discovery
node_search_api -->|"delegates search"| node_search_service
node_search_service -->|"queries providers"| node_search_providers
node_search_service -->|"triages results"| node_triage
node_discovery -->|"resolves sources"| node_source_extractors
node_job_api -->|"builds preview"| node_preview
node_preview -->|"fetches preview text"| node_source_extractors
node_reader -->|"reviews and reads"| node_job_page
node_job_page -->|"confirms selection"| node_job_api
node_job_api -->|"starts compilation"| node_job_runner
node_job_runner -->|"updates job state"| node_job_repository
node_job_repository -->|"reads and writes"| node_sqlite
node_backend_app -->|"initializes database"| node_sqlite
node_job_runner -->|"fetches selected content"| node_source_extractors
node_job_runner -->|"uses transcript cache"| node_transcript_cache
node_job_runner -->|"applies cleanup roles"| node_cleanup
node_cleanup -->|"requests polish"| node_llm
node_job_runner -->|"transcribes episodes"| node_stt
node_job_runner -->|"compiles selected text"| node_compiler
node_job_runner -->|"renders book"| node_renderer
node_reader -->|"downloads EPUB or Markdown"| node_download
node_job_page -->|"stores history locally"| node_browser_history

click node_workspace "https://github.com/wl2bd/thothly/blob/main/frontend/components/compose.tsx"
click node_job_page "https://github.com/wl2bd/thothly/blob/main/frontend/app/jobs/%5Bid%5D/page.tsx"
click node_browser_history "https://github.com/wl2bd/thothly/blob/main/frontend/lib/history.ts"
click node_search_api "https://github.com/wl2bd/thothly/blob/main/backend/app/api/search.py"
click node_search_service "https://github.com/wl2bd/thothly/blob/main/backend/app/search/service.py"
click node_search_providers "https://github.com/wl2bd/thothly/tree/main/backend/app/search"
click node_triage "https://github.com/wl2bd/thothly/blob/main/backend/app/search/triage.py"
click node_job_api "https://github.com/wl2bd/thothly/blob/main/backend/app/jobs/router.py"
click node_discovery "https://github.com/wl2bd/thothly/blob/main/backend/app/sources/discovery.py"
click node_source_extractors "https://github.com/wl2bd/thothly/tree/main/backend/app/sources"
click node_preview "https://github.com/wl2bd/thothly/blob/main/backend/app/jobs/preview.py"
click node_job_runner "https://github.com/wl2bd/thothly/blob/main/backend/app/jobs/runner.py"
click node_job_repository "https://github.com/wl2bd/thothly/blob/main/backend/app/jobs/repository.py"
click node_sqlite "https://github.com/wl2bd/thothly/blob/main/backend/app/core/database.py"
click node_transcript_cache "https://github.com/wl2bd/thothly/blob/main/backend/app/sources/transcript_cache.py"
click node_cleanup "https://github.com/wl2bd/thothly/blob/main/backend/app/pipeline/cleanup.py"
click node_llm "https://github.com/wl2bd/thothly/blob/main/backend/app/pipeline/llm.py"
click node_stt "https://github.com/wl2bd/thothly/blob/main/backend/app/pipeline/transcribe.py"
click node_compiler "https://github.com/wl2bd/thothly/blob/main/backend/app/pipeline/compiler.py"
click node_renderer "https://github.com/wl2bd/thothly/blob/main/backend/app/render/epub.py"
click node_backend_app "https://github.com/wl2bd/thothly/blob/main/backend/app/main.py"
click node_download "https://github.com/wl2bd/thothly/blob/main/backend/app/jobs/router.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_workspace,node_job_page,node_browser_history,node_reader toneBlue
class node_search_service,node_search_providers,node_triage,node_discovery toneAmber
class node_job_api,node_preview,node_job_runner,node_job_repository,node_download toneMint
class node_source_extractors,node_transcript_cache,node_cleanup,node_llm,node_stt,node_compiler,node_renderer toneRose
class node_search_api,node_sqlite,node_backend_app toneIndigo
```

## Components

### Browser workspace

- [Compose workspace](https://github.com/wl2bd/thothly/blob/main/frontend/components/compose.tsx): `frontend/components/compose.tsx`
- [Compilation reader](https://github.com/wl2bd/thothly/blob/main/frontend/app/jobs/%5Bid%5D/page.tsx): `frontend/app/jobs/[id]/page.tsx`
- [Browser history](https://github.com/wl2bd/thothly/blob/main/frontend/lib/history.ts): `frontend/lib/history.ts`

### Search and discovery

- [Search aggregation](https://github.com/wl2bd/thothly/blob/main/backend/app/search/service.py): `backend/app/search/service.py`
- [Search providers](https://github.com/wl2bd/thothly/tree/main/backend/app/search): `backend/app/search`
- [Search triage](https://github.com/wl2bd/thothly/blob/main/backend/app/search/triage.py): `backend/app/search/triage.py`
- [Source discovery](https://github.com/wl2bd/thothly/blob/main/backend/app/sources/discovery.py): `backend/app/sources/discovery.py`

### Compilation jobs

- [Job routes](https://github.com/wl2bd/thothly/blob/main/backend/app/jobs/router.py): `backend/app/jobs/router.py`
- [Review previews](https://github.com/wl2bd/thothly/blob/main/backend/app/jobs/preview.py): `backend/app/jobs/preview.py`
- [Job runner](https://github.com/wl2bd/thothly/blob/main/backend/app/jobs/runner.py): `backend/app/jobs/runner.py`
- [Job repository](https://github.com/wl2bd/thothly/blob/main/backend/app/jobs/repository.py): `backend/app/jobs/repository.py`
- [Book download](https://github.com/wl2bd/thothly/blob/main/backend/app/jobs/router.py): `backend/app/jobs/router.py`

### Content processing

- [Source extractors](https://github.com/wl2bd/thothly/tree/main/backend/app/sources): `backend/app/sources`
- [Transcript caches](https://github.com/wl2bd/thothly/blob/main/backend/app/sources/transcript_cache.py): `backend/app/sources/transcript_cache.py`
- [Text cleanup](https://github.com/wl2bd/thothly/blob/main/backend/app/pipeline/cleanup.py): `backend/app/pipeline/cleanup.py`
- [LLM client](https://github.com/wl2bd/thothly/blob/main/backend/app/pipeline/llm.py): `backend/app/pipeline/llm.py`
- [Podcast transcription](https://github.com/wl2bd/thothly/blob/main/backend/app/pipeline/transcribe.py): `backend/app/pipeline/transcribe.py`
- [Content compiler](https://github.com/wl2bd/thothly/blob/main/backend/app/pipeline/compiler.py): `backend/app/pipeline/compiler.py`
- [EPUB and cover renderer](https://github.com/wl2bd/thothly/blob/main/backend/app/render/epub.py): `backend/app/render/epub.py`

### Persistence and API

- [Search API](https://github.com/wl2bd/thothly/blob/main/backend/app/api/search.py): `backend/app/api/search.py`
- [SQLite database](https://github.com/wl2bd/thothly/blob/main/backend/app/core/database.py): `backend/app/core/database.py`
- [Backend application](https://github.com/wl2bd/thothly/blob/main/backend/app/main.py): `backend/app/main.py`

### Other components

- **User**

## Connections

- User → Compose workspace: searches or pastes
- Compose workspace → Search API: searches
- Compose workspace → Job routes: starts job
- Job routes → Job repository: creates and reads (evidence: `backend/app/jobs/router.py`)
- Job routes → Source discovery: starts discovery (evidence: `backend/app/jobs/router.py`)
- Search API → Search aggregation: delegates search (evidence: `backend/app/api/search.py`)
- Search aggregation → Search providers: queries providers (evidence: `backend/app/search/service.py`)
- Search aggregation → Search triage: triages results (evidence: `backend/app/search/service.py`)
- Source discovery → Source extractors: resolves sources (evidence: `backend/app/jobs/phases.py`)
- Job routes → Review previews: builds preview (evidence: `backend/app/jobs/router.py`)
- Review previews → Source extractors: fetches preview text (evidence: `backend/app/jobs/preview.py`)
- User → Compilation reader: reviews and reads
- Compilation reader → Job routes: confirms selection
- Job routes → Job runner: starts compilation (evidence: `backend/app/jobs/router.py`)
- Job runner → Job repository: updates job state (evidence: `backend/app/jobs/runner.py`)
- Job repository → SQLite database: reads and writes (evidence: `backend/app/jobs/repository.py`)
- Backend application → SQLite database: initializes database (evidence: `backend/app/main.py`)
- Job runner → Source extractors: fetches selected content (evidence: `backend/app/jobs/runner.py`)
- Job runner → Transcript caches: uses transcript cache (evidence: `backend/app/jobs/runner.py`)
- Job runner → Text cleanup: applies cleanup roles (evidence: `backend/app/jobs/runner.py`)
- Text cleanup → LLM client: requests polish (evidence: `backend/app/pipeline/llm.py`)
- Job runner → Podcast transcription: transcribes episodes (evidence: `backend/app/jobs/runner.py`)
- Job runner → Content compiler: compiles selected text (evidence: `backend/app/jobs/runner.py`)
- Job runner → EPUB and cover renderer: renders book (evidence: `backend/app/jobs/runner.py`)
- User → Book download: downloads EPUB or Markdown
- Compilation reader → Browser history: stores history locally
