# kishormorol/researchscope architecture

> Architecture diagram of the GitHub repository kishormorol/researchscope, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kishormorol/researchscope
- Repository: https://github.com/kishormorol/researchscope

## Overview

ResearchScope combines a daily paper-ingestion and enrichment pipeline, a FastAPI-backed searchable catalogue, and a lightweight local CLI. The primary workflow gathers papers from scholarly sources, enriches and analyzes them, then publishes data to the site and storage backends for researchers to search and explore. The README also documents venue recommenders, deadlines, author intelligence, and paper chat. Some branches are shown from README documentation because their source was not sampled; unsampled implementation details and wiring are not inferred.

Main entry points: [`src/pipeline.py`](https://github.com/kishormorol/researchscope/blob/main/src/pipeline.py) (Pipeline runner), [`backend/app/main.py`](https://github.com/kishormorol/researchscope/blob/main/backend/app/main.py) (REST API), [`researchscope/cli.py`](https://github.com/kishormorol/researchscope/blob/main/researchscope/cli.py) (ResearchScope CLI), [`backend/app/database.py`](https://github.com/kishormorol/researchscope/blob/main/backend/app/database.py) (API database)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_pipeline["Ingestion pipeline"]
  node_pipeline_runner["Pipeline runner<br/>[pipeline.py]"]
  node_connectors["Source connectors"]
  node_normalization["Paper schema<br/>[schema.py]"]
  node_dedup["Deduplication<br/>[deduplicator.py]"]
  node_tagging["Topic tagging<br/>[tagger.py]"]
  node_difficulty["Reading difficulty<br/>[assessor.py]"]
end

subgraph group_analysis["Research intelligence"]
  node_scoring["Paper scoring<br/>[scorer.py]"]
  node_content["Paper enrichment<br/>[generator.py]"]
  node_clustering["Topic clustering<br/>[clusterer.py]"]
  node_gaps["Research gaps<br/>[gap_extractor.py]"]
  node_aggregation["Author profiles<br/>[aggregator.py]"]
  node_venue_recommenders["Venue recommenders"]
end

subgraph group_delivery["Publishing and storage"]
  node_sitegen["Site generation<br/>[generator.py]"]
  node_railway_store[("Railway catalogue<br/>[railway_store.py]")]
  node_hf_dataset["Hugging Face dataset<br/>[hf_dataset.py]"]
end

subgraph group_api["API and paper chat"]
  node_api["REST API<br/>[main.py]"]
  node_api_routers["Paper and search routes"]
  node_auth_favourites["Accounts and favourites"]
  node_chat_routes["Chat and document routes"]
  node_document_service["PDF preparation"]
  node_chat_service["Paper chat<br/>[chat_service.py]"]
  node_retrieval["Evidence retrieval"]
  node_providers["Model providers"]
  node_api_database[("API database<br/>[database.py]")]
end

subgraph group_cli["Local CLI"]
  node_cli["ResearchScope CLI<br/>[cli.py]"]
  node_cli_collectors["CLI collectors"]
  node_cli_store[("Local paper store<br/>[store.py]")]
  node_cli_analysis["Saved-paper analysis"]
end

node_researcher(("Researcher"))
node_sources["Scholarly sources"]

node_sources -->|"provides papers"| node_connectors
node_pipeline_runner -->|"fetches papers"| node_connectors
node_pipeline_runner -->|"deduplicates"| node_dedup
node_pipeline_runner -->|"tags papers"| node_tagging
node_pipeline_runner -->|"assesses difficulty"| node_difficulty
node_pipeline_runner -->|"scores papers"| node_scoring
node_pipeline_runner -->|"enriches papers"| node_content
node_pipeline_runner -->|"groups topics"| node_clustering
node_pipeline_runner -->|"extracts gaps"| node_gaps
node_pipeline_runner -->|"builds profiles"| node_aggregation
node_pipeline_runner -->|"generates site data"| node_sitegen
node_pipeline_runner -->|"syncs papers"| node_railway_store
node_pipeline_runner -->|"syncs dataset"| node_hf_dataset
node_researcher -->|"uses"| node_api
node_api -->|"registers routes"| node_api_routers
node_api -->|"registers routes"| node_auth_favourites
node_api -->|"registers routes"| node_chat_routes
node_api_routers -->|"queries catalogue"| node_api_database
node_auth_favourites -->|"reads and writes"| node_api_database
node_chat_routes -->|"prepares documents"| node_document_service
node_chat_routes -->|"handles chat"| node_chat_service
node_chat_service -->|"retrieves evidence"| node_retrieval
node_chat_service -->|"requests responses"| node_providers
node_document_service -->|"stores document data"| node_api_database
node_retrieval -->|"searches chunks"| node_api_database
node_researcher -->|"gets venue matches"| node_venue_recommenders
node_researcher -->|"explores gaps"| node_gaps
node_researcher -->|"explores researchers"| node_aggregation
node_researcher -->|"runs commands"| node_cli
node_cli -->|"searches sources"| node_cli_collectors
node_cli -->|"saves papers"| node_cli_store
node_cli -->|"analyzes saved papers"| node_cli_analysis
node_cli_collectors -->|"queries"| node_sources

click node_pipeline_runner "https://github.com/kishormorol/researchscope/blob/main/src/pipeline.py"
click node_connectors "https://github.com/kishormorol/researchscope/tree/main/src/connectors"
click node_normalization "https://github.com/kishormorol/researchscope/blob/main/src/normalization/schema.py"
click node_dedup "https://github.com/kishormorol/researchscope/blob/main/src/dedup/deduplicator.py"
click node_tagging "https://github.com/kishormorol/researchscope/blob/main/src/tagging/tagger.py"
click node_difficulty "https://github.com/kishormorol/researchscope/blob/main/src/difficulty/assessor.py"
click node_scoring "https://github.com/kishormorol/researchscope/blob/main/src/scoring/scorer.py"
click node_content "https://github.com/kishormorol/researchscope/blob/main/src/content/generator.py"
click node_clustering "https://github.com/kishormorol/researchscope/blob/main/src/clustering/clusterer.py"
click node_gaps "https://github.com/kishormorol/researchscope/blob/main/src/gaps/gap_extractor.py"
click node_aggregation "https://github.com/kishormorol/researchscope/blob/main/src/aggregation/aggregator.py"
click node_venue_recommenders "https://github.com/kishormorol/researchscope/tree/main/src/sitegen"
click node_sitegen "https://github.com/kishormorol/researchscope/blob/main/src/sitegen/generator.py"
click node_railway_store "https://github.com/kishormorol/researchscope/blob/main/src/storage/railway_store.py"
click node_hf_dataset "https://github.com/kishormorol/researchscope/blob/main/src/storage/hf_dataset.py"
click node_api "https://github.com/kishormorol/researchscope/blob/main/backend/app/main.py"
click node_api_routers "https://github.com/kishormorol/researchscope/tree/main/backend/app/routers"
click node_auth_favourites "https://github.com/kishormorol/researchscope/tree/main/backend/app/routers"
click node_chat_routes "https://github.com/kishormorol/researchscope/tree/main/backend/app/routers"
click node_document_service "https://github.com/kishormorol/researchscope/blob/main/backend/app/services/document_service.py"
click node_chat_service "https://github.com/kishormorol/researchscope/blob/main/backend/app/services/chat_service.py"
click node_retrieval "https://github.com/kishormorol/researchscope/blob/main/backend/app/services/retrieval_service.py"
click node_providers "https://github.com/kishormorol/researchscope/blob/main/backend/app/services/provider_service.py"
click node_api_database "https://github.com/kishormorol/researchscope/blob/main/backend/app/database.py"
click node_cli "https://github.com/kishormorol/researchscope/blob/main/researchscope/cli.py"
click node_cli_collectors "https://github.com/kishormorol/researchscope/tree/main/researchscope/collectors"
click node_cli_store "https://github.com/kishormorol/researchscope/blob/main/researchscope/storage/store.py"
click node_cli_analysis "https://github.com/kishormorol/researchscope/tree/main/researchscope/analysis"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_pipeline_runner,node_connectors,node_normalization,node_dedup,node_tagging,node_difficulty toneBlue
class node_scoring,node_content,node_clustering,node_gaps,node_aggregation,node_venue_recommenders toneAmber
class node_sitegen,node_railway_store,node_hf_dataset toneMint
class node_api,node_api_routers,node_auth_favourites,node_chat_routes,node_document_service,node_chat_service,node_retrieval,node_providers,node_api_database toneRose
class node_cli,node_cli_collectors,node_cli_store,node_cli_analysis,node_researcher,node_sources toneIndigo
```

## Components

### Ingestion pipeline

- [Pipeline runner](https://github.com/kishormorol/researchscope/blob/main/src/pipeline.py): `src/pipeline.py`
- [Source connectors](https://github.com/kishormorol/researchscope/tree/main/src/connectors): `src/connectors`
- [Paper schema](https://github.com/kishormorol/researchscope/blob/main/src/normalization/schema.py): `src/normalization/schema.py`
- [Deduplication](https://github.com/kishormorol/researchscope/blob/main/src/dedup/deduplicator.py): `src/dedup/deduplicator.py`
- [Topic tagging](https://github.com/kishormorol/researchscope/blob/main/src/tagging/tagger.py): `src/tagging/tagger.py`
- [Reading difficulty](https://github.com/kishormorol/researchscope/blob/main/src/difficulty/assessor.py): `src/difficulty/assessor.py`

### Research intelligence

- [Paper scoring](https://github.com/kishormorol/researchscope/blob/main/src/scoring/scorer.py): `src/scoring/scorer.py`
- [Paper enrichment](https://github.com/kishormorol/researchscope/blob/main/src/content/generator.py): `src/content/generator.py`
- [Topic clustering](https://github.com/kishormorol/researchscope/blob/main/src/clustering/clusterer.py): `src/clustering/clusterer.py`
- [Research gaps](https://github.com/kishormorol/researchscope/blob/main/src/gaps/gap_extractor.py): `src/gaps/gap_extractor.py`
- [Author profiles](https://github.com/kishormorol/researchscope/blob/main/src/aggregation/aggregator.py): `src/aggregation/aggregator.py`
- [Venue recommenders](https://github.com/kishormorol/researchscope/tree/main/src/sitegen): `src/sitegen`

### Publishing and storage

- [Site generation](https://github.com/kishormorol/researchscope/blob/main/src/sitegen/generator.py): `src/sitegen/generator.py`
- [Railway catalogue](https://github.com/kishormorol/researchscope/blob/main/src/storage/railway_store.py): `src/storage/railway_store.py`
- [Hugging Face dataset](https://github.com/kishormorol/researchscope/blob/main/src/storage/hf_dataset.py): `src/storage/hf_dataset.py`

### API and paper chat

- [REST API](https://github.com/kishormorol/researchscope/blob/main/backend/app/main.py): `backend/app/main.py`
- [Paper and search routes](https://github.com/kishormorol/researchscope/tree/main/backend/app/routers): `backend/app/routers`
- [Accounts and favourites](https://github.com/kishormorol/researchscope/tree/main/backend/app/routers): `backend/app/routers`
- [Chat and document routes](https://github.com/kishormorol/researchscope/tree/main/backend/app/routers): `backend/app/routers`
- [PDF preparation](https://github.com/kishormorol/researchscope/blob/main/backend/app/services/document_service.py): `backend/app/services/document_service.py`
- [Paper chat](https://github.com/kishormorol/researchscope/blob/main/backend/app/services/chat_service.py): `backend/app/services/chat_service.py`
- [Evidence retrieval](https://github.com/kishormorol/researchscope/blob/main/backend/app/services/retrieval_service.py): `backend/app/services/retrieval_service.py`
- [Model providers](https://github.com/kishormorol/researchscope/blob/main/backend/app/services/provider_service.py): `backend/app/services/provider_service.py`
- [API database](https://github.com/kishormorol/researchscope/blob/main/backend/app/database.py): `backend/app/database.py`

### Local CLI

- [ResearchScope CLI](https://github.com/kishormorol/researchscope/blob/main/researchscope/cli.py): `researchscope/cli.py`
- [CLI collectors](https://github.com/kishormorol/researchscope/tree/main/researchscope/collectors): `researchscope/collectors`
- [Local paper store](https://github.com/kishormorol/researchscope/blob/main/researchscope/storage/store.py): `researchscope/storage/store.py`
- [Saved-paper analysis](https://github.com/kishormorol/researchscope/tree/main/researchscope/analysis): `researchscope/analysis`

### Other components

- **Researcher**
- **Scholarly sources**

## Connections

- Scholarly sources → Source connectors: provides papers (evidence: `README.md`)
- Pipeline runner → Source connectors: fetches papers (evidence: `src/pipeline.py`)
- Pipeline runner → Deduplication: deduplicates (evidence: `README.md`)
- Pipeline runner → Topic tagging: tags papers (evidence: `README.md`)
- Pipeline runner → Reading difficulty: assesses difficulty (evidence: `README.md`)
- Pipeline runner → Paper scoring: scores papers (evidence: `src/pipeline.py`)
- Pipeline runner → Paper enrichment: enriches papers (evidence: `src/pipeline.py`)
- Pipeline runner → Topic clustering: groups topics (evidence: `README.md`)
- Pipeline runner → Research gaps: extracts gaps (evidence: `README.md`)
- Pipeline runner → Author profiles: builds profiles (evidence: `README.md`)
- Pipeline runner → Site generation: generates site data (evidence: `README.md`)
- Pipeline runner → Railway catalogue: syncs papers (evidence: `README.md`)
- Pipeline runner → Hugging Face dataset: syncs dataset (evidence: `README.md`)
- Researcher → REST API: uses (evidence: `README.md`)
- REST API → Paper and search routes: registers routes (evidence: `backend/app/main.py`)
- REST API → Accounts and favourites: registers routes (evidence: `backend/app/main.py`)
- REST API → Chat and document routes: registers routes (evidence: `backend/app/main.py`)
- Paper and search routes → API database: queries catalogue (evidence: `backend/app/routers/papers.py`)
- Accounts and favourites → API database: reads and writes (evidence: `backend/app/routers/favourites.py`)
- Chat and document routes → PDF preparation: prepares documents (evidence: `backend/app/routers/paper_documents.py`)
- Chat and document routes → Paper chat: handles chat (evidence: `backend/app/routers/chat.py`)
- Paper chat → Evidence retrieval: retrieves evidence (evidence: `backend/app/services/chat_service.py`)
- Paper chat → Model providers: requests responses (evidence: `backend/app/services/chat_service.py`)
- PDF preparation → API database: stores document data (evidence: `backend/app/services/document_service.py`)
- Evidence retrieval → API database: searches chunks (evidence: `backend/app/services/retrieval_service.py`)
- Researcher → Venue recommenders: gets venue matches (evidence: `README.md`)
- Researcher → Research gaps: explores gaps (evidence: `README.md`)
- Researcher → Author profiles: explores researchers (evidence: `README.md`)
- Researcher → ResearchScope CLI: runs commands (evidence: `README.md`)
- ResearchScope CLI → CLI collectors: searches sources (evidence: `researchscope/cli.py`)
- ResearchScope CLI → Local paper store: saves papers (evidence: `researchscope/cli.py`)
- ResearchScope CLI → Saved-paper analysis: analyzes saved papers (evidence: `README.md`)
- CLI collectors → Scholarly sources: queries (evidence: `README.md`)