# l34n/ai-researcher architecture

> Architecture diagram of the GitHub repository l34n/ai-researcher, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/l34n/ai-researcher
- Repository: https://github.com/l34n/ai-researcher

## Overview

AI Researcher is a self-hosted AI news and trends dashboard. Users trigger or schedule a refresh, which collects items from configured sources, enriches and clusters them into stories, judges readiness, and produces deep-research pages and a daily brief for dashboard browsing. The graph preserves those distinct stages plus search, saved items, and source/run status. Pipeline and web wiring are based on sampled excerpts and README documentation; unsampled module internals and uncertain links are omitted.

Main entry points: [`src/ai_researcher/pipeline.py`](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/pipeline.py) (Refresh Pipeline), [`src/ai_researcher/db.py`](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/db.py) (SQLite Store), [`src/ai_researcher/web/app.py`](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/web/app.py) (Web Dashboard), [`src/ai_researcher/enrich/chat.py`](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/enrich/chat.py) (Chat Routing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Collection"]
  node_cli["CLI<br/>[cli.py]"]
  node_pipeline["Refresh Pipeline<br/>[pipeline.py]"]
  node_connectors["Source Connectors"]
  node_article["Article Fetching<br/>[article.py]"]
end

subgraph group_analysis["Story Analysis"]
  node_enrichment["Item Enrichment"]
  node_chat["Chat Routing<br/>[chat.py]"]
  node_embeddings["Embeddings<br/>[embed.py]"]
  node_relevance["Relevance Filter<br/>[relevance.py]"]
  node_clustering["Story Clustering<br/>[cluster.py]"]
  node_ranking["Trend Ranking<br/>[score.py]"]
  node_freshness["Freshness Checks<br/>[freshness.py]"]
  node_judgment["Readiness Judgment<br/>[judge.py]"]
end

subgraph group_research["Research Outputs"]
  node_deep_research["Deep Research<br/>[wiki.py]"]
  node_brief["Daily Brief<br/>[brief.py]"]
end

subgraph group_serving["Dashboard"]
  node_web_app["Web Dashboard<br/>[app.py]"]
  node_queries["Dashboard Queries<br/>[queries.py]"]
  node_web_client["Dashboard Client<br/>[app.js]"]
end

subgraph group_runtime["Runtime State"]
  node_database[("SQLite Store<br/>[db.py]")]
  node_progress["Run Progress<br/>[progress.py]"]
end

node_user(("Researcher"))
node_sources(("News Sources"))
node_ollama["Ollama"]
node_cloud_models["Cloud Models"]

node_user -->|"runs commands"| node_cli
node_cli -->|"starts refresh"| node_pipeline
node_user -->|"browses dashboard"| node_web_app
node_web_client -->|"requests actions"| node_web_app
node_web_app -->|"starts refresh"| node_pipeline
node_pipeline -->|"fetches sources"| node_connectors
node_sources -->|"provides feeds"| node_connectors
node_pipeline -->|"fetches article text"| node_article
node_article -->|"requests pages"| node_sources
node_pipeline -->|"stores items"| node_database
node_pipeline -->|"enriches items"| node_enrichment
node_enrichment -->|"requests model text"| node_chat
node_enrichment -->|"embeds items"| node_embeddings
node_chat -.->|"routes locally"| node_ollama
node_chat -.->|"routes optionally"| node_cloud_models
node_pipeline -->|"filters items"| node_relevance
node_pipeline -->|"checks freshness"| node_freshness
node_pipeline -->|"builds stories"| node_clustering
node_pipeline -->|"ranks stories"| node_ranking
node_pipeline -->|"judges readiness"| node_judgment
node_pipeline -->|"researches ready stories"| node_deep_research
node_deep_research -->|"generates pages"| node_chat
node_pipeline -->|"generates brief"| node_brief
node_brief -->|"generates summary"| node_chat
node_enrichment -->|"writes enrichment"| node_database
node_clustering -->|"writes clusters"| node_database
node_ranking -->|"stores scores"| node_database
node_judgment -->|"stores judgments"| node_database
node_deep_research -->|"stores research"| node_database
node_brief -->|"stores brief"| node_database
node_web_app -->|"loads dashboard data"| node_queries
node_queries -->|"reads records"| node_database
node_web_app -->|"updates saved items"| node_database
node_pipeline -->|"reports progress"| node_progress
node_web_app -->|"serves run status"| node_progress

click node_cli "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/cli.py"
click node_pipeline "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/pipeline.py"
click node_connectors "https://github.com/l34n/ai-researcher/tree/main/src/ai_researcher/connectors"
click node_article "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/connectors/article.py"
click node_database "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/db.py"
click node_enrichment "https://github.com/l34n/ai-researcher/tree/main/src/ai_researcher/enrich"
click node_chat "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/enrich/chat.py"
click node_embeddings "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/enrich/embed.py"
click node_relevance "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/enrich/relevance.py"
click node_clustering "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/trends/cluster.py"
click node_ranking "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/trends/score.py"
click node_freshness "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/trends/freshness.py"
click node_judgment "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/enrich/judge.py"
click node_deep_research "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/research/wiki.py"
click node_brief "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/trends/brief.py"
click node_web_app "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/web/app.py"
click node_queries "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/web/queries.py"
click node_web_client "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/web/static/app.js"
click node_progress "https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/progress.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_pipeline,node_connectors,node_article toneBlue
class node_enrichment,node_chat,node_embeddings,node_relevance,node_clustering,node_ranking,node_freshness,node_judgment toneAmber
class node_deep_research,node_brief toneMint
class node_web_app,node_queries,node_web_client toneRose
class node_database,node_progress,node_user,node_sources,node_ollama,node_cloud_models toneIndigo
```

## Components

### Collection

- [CLI](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/cli.py): `src/ai_researcher/cli.py`
- [Refresh Pipeline](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/pipeline.py): `src/ai_researcher/pipeline.py`
- [Source Connectors](https://github.com/l34n/ai-researcher/tree/main/src/ai_researcher/connectors): `src/ai_researcher/connectors`
- [Article Fetching](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/connectors/article.py): `src/ai_researcher/connectors/article.py`

### Story Analysis

- [Item Enrichment](https://github.com/l34n/ai-researcher/tree/main/src/ai_researcher/enrich): `src/ai_researcher/enrich`
- [Chat Routing](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/enrich/chat.py): `src/ai_researcher/enrich/chat.py`
- [Embeddings](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/enrich/embed.py): `src/ai_researcher/enrich/embed.py`
- [Relevance Filter](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/enrich/relevance.py): `src/ai_researcher/enrich/relevance.py`
- [Story Clustering](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/trends/cluster.py): `src/ai_researcher/trends/cluster.py`
- [Trend Ranking](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/trends/score.py): `src/ai_researcher/trends/score.py`
- [Freshness Checks](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/trends/freshness.py): `src/ai_researcher/trends/freshness.py`
- [Readiness Judgment](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/enrich/judge.py): `src/ai_researcher/enrich/judge.py`

### Research Outputs

- [Deep Research](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/research/wiki.py): `src/ai_researcher/research/wiki.py`
- [Daily Brief](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/trends/brief.py): `src/ai_researcher/trends/brief.py`

### Dashboard

- [Web Dashboard](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/web/app.py): `src/ai_researcher/web/app.py`
- [Dashboard Queries](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/web/queries.py): `src/ai_researcher/web/queries.py`
- [Dashboard Client](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/web/static/app.js): `src/ai_researcher/web/static/app.js`

### Runtime State

- [SQLite Store](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/db.py): `src/ai_researcher/db.py`
- [Run Progress](https://github.com/l34n/ai-researcher/blob/main/src/ai_researcher/progress.py): `src/ai_researcher/progress.py`

### Other components

- **Researcher**
- **News Sources**
- **Ollama**
- **Cloud Models**

## Connections

- Researcher → CLI: runs commands
- CLI → Refresh Pipeline: starts refresh
- Researcher → Web Dashboard: browses dashboard
- Dashboard Client → Web Dashboard: requests actions
- Web Dashboard → Refresh Pipeline: starts refresh
- Refresh Pipeline → Source Connectors: fetches sources
- News Sources → Source Connectors: provides feeds
- Refresh Pipeline → Article Fetching: fetches article text
- Article Fetching → News Sources: requests pages
- Refresh Pipeline → SQLite Store: stores items
- Refresh Pipeline → Item Enrichment: enriches items
- Item Enrichment → Chat Routing: requests model text
- Item Enrichment → Embeddings: embeds items
- Chat Routing → Ollama: routes locally
- Chat Routing → Cloud Models: routes optionally
- Refresh Pipeline → Relevance Filter: filters items
- Refresh Pipeline → Freshness Checks: checks freshness
- Refresh Pipeline → Story Clustering: builds stories
- Refresh Pipeline → Trend Ranking: ranks stories
- Refresh Pipeline → Readiness Judgment: judges readiness
- Refresh Pipeline → Deep Research: researches ready stories
- Deep Research → Chat Routing: generates pages
- Refresh Pipeline → Daily Brief: generates brief
- Daily Brief → Chat Routing: generates summary
- Item Enrichment → SQLite Store: writes enrichment
- Story Clustering → SQLite Store: writes clusters
- Trend Ranking → SQLite Store: stores scores
- Readiness Judgment → SQLite Store: stores judgments
- Deep Research → SQLite Store: stores research
- Daily Brief → SQLite Store: stores brief
- Web Dashboard → Dashboard Queries: loads dashboard data
- Dashboard Queries → SQLite Store: reads records
- Web Dashboard → SQLite Store: updates saved items
- Refresh Pipeline → Run Progress: reports progress
- Web Dashboard → Run Progress: serves run status