# rpc1418/document-retrieval-system architecture

> Architecture diagram of the GitHub repository rpc1418/document-retrieval-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rpc1418/document-retrieval-system
- Repository: https://github.com/rpc1418/document-retrieval-system

## Overview

This is a Flask document-retrieval service: users query indexed news articles and receive similarity-ranked snippets. The main flow applies per-user request limits, checks SQLite cache, then searches stored documents with TF-IDF and cosine similarity. Separately, a background thread fetches articles from three news sites, parses them, and stores new documents. Health checks and a standalone search script are also represented. The runtime is concentrated in app.py; README-described behavior is included, but unsampled modules and deployment details are not expanded.

Main entry points: [`cache.db`](https://github.com/rpc1418/document-retrieval-system/blob/main/cache.db) (Document store), [`app.py`](https://github.com/rpc1418/document-retrieval-system/blob/main/app.py) (Search endpoint), [`fetchCode.py`](https://github.com/rpc1418/document-retrieval-system/blob/main/fetchCode.py) (Standalone search)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["HTTP API"]
  node_search_api["Search endpoint<br/>[app.py]"]
  node_health_api["Health endpoint<br/>[app.py]"]
  node_rate_limit["Request limiter<br/>[app.py]"]
end

subgraph group_retrieval["Document Retrieval"]
  node_search_engine["Ranked search<br/>[app.py]"]
  node_similarity["TF-IDF scoring<br/>[app.py]"]
  node_standalone_search["Standalone search<br/>[fetchCode.py]"]
end

subgraph group_ingestion["News Ingestion"]
  node_scraper_thread["Scraper thread<br/>[app.py]"]
  node_article_fetch["Article fetching<br/>[app.py]"]
  node_article_parser["Article parsing<br/>[app.py]"]
  node_duplicate_check["Title deduplication<br/>[app.py]"]
  node_article_write["Article storage<br/>[app.py]"]
end

subgraph group_state["Persistent State"]
  node_cache_ops["Result cache<br/>[app.py]"]
  node_document_store[("Document store<br/>[cache.db]")]
end

node_user(("Chat client"))
node_news_sites["Trusted news sites"]

node_user -->|"queries"| node_search_api
node_user -->|"checks health"| node_health_api
node_search_api -->|"checks limit"| node_rate_limit
node_rate_limit -->|"reads/writes counts"| node_document_store
node_search_api -->|"checks cache"| node_cache_ops
node_cache_ops -->|"reads/writes cache"| node_document_store
node_search_api -->|"runs search"| node_search_engine
node_search_engine -->|"reads documents"| node_document_store
node_search_engine -->|"scores matches"| node_similarity
node_search_api -->|"stores results"| node_cache_ops
node_standalone_search -->|"reads documents"| node_document_store
node_standalone_search -->|"scores matches"| node_similarity
node_scraper_thread -->|"runs scraper"| node_article_fetch
node_article_fetch -->|"fetches pages"| node_news_sites
node_article_fetch -->|"parses responses"| node_article_parser
node_article_parser -->|"checks titles"| node_duplicate_check
node_duplicate_check -->|"checks existing"| node_document_store
node_article_parser -->|"passes articles"| node_article_write
node_article_write -->|"inserts articles"| node_document_store

click node_search_api "https://github.com/rpc1418/document-retrieval-system/blob/main/app.py"
click node_health_api "https://github.com/rpc1418/document-retrieval-system/blob/main/app.py"
click node_rate_limit "https://github.com/rpc1418/document-retrieval-system/blob/main/app.py"
click node_cache_ops "https://github.com/rpc1418/document-retrieval-system/blob/main/app.py"
click node_document_store "https://github.com/rpc1418/document-retrieval-system/blob/main/cache.db"
click node_search_engine "https://github.com/rpc1418/document-retrieval-system/blob/main/app.py"
click node_similarity "https://github.com/rpc1418/document-retrieval-system/blob/main/app.py"
click node_standalone_search "https://github.com/rpc1418/document-retrieval-system/blob/main/fetchCode.py"
click node_scraper_thread "https://github.com/rpc1418/document-retrieval-system/blob/main/app.py"
click node_article_fetch "https://github.com/rpc1418/document-retrieval-system/blob/main/app.py"
click node_article_parser "https://github.com/rpc1418/document-retrieval-system/blob/main/app.py"
click node_duplicate_check "https://github.com/rpc1418/document-retrieval-system/blob/main/app.py"
click node_article_write "https://github.com/rpc1418/document-retrieval-system/blob/main/app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_search_api,node_health_api,node_rate_limit,node_user toneBlue
class node_search_engine,node_similarity,node_standalone_search toneAmber
class node_scraper_thread,node_article_fetch,node_article_parser,node_duplicate_check,node_article_write toneMint
class node_cache_ops,node_document_store toneRose
class node_news_sites toneIndigo
```

## Components

### HTTP API

- [Search endpoint](https://github.com/rpc1418/document-retrieval-system/blob/main/app.py): `app.py`
- [Health endpoint](https://github.com/rpc1418/document-retrieval-system/blob/main/app.py): `app.py`
- [Request limiter](https://github.com/rpc1418/document-retrieval-system/blob/main/app.py): `app.py`

### Document Retrieval

- [Ranked search](https://github.com/rpc1418/document-retrieval-system/blob/main/app.py): `app.py`
- [TF-IDF scoring](https://github.com/rpc1418/document-retrieval-system/blob/main/app.py): `app.py`
- [Standalone search](https://github.com/rpc1418/document-retrieval-system/blob/main/fetchCode.py): `fetchCode.py`

### News Ingestion

- [Scraper thread](https://github.com/rpc1418/document-retrieval-system/blob/main/app.py): `app.py`
- [Article fetching](https://github.com/rpc1418/document-retrieval-system/blob/main/app.py): `app.py`
- [Article parsing](https://github.com/rpc1418/document-retrieval-system/blob/main/app.py): `app.py`
- [Title deduplication](https://github.com/rpc1418/document-retrieval-system/blob/main/app.py): `app.py`
- [Article storage](https://github.com/rpc1418/document-retrieval-system/blob/main/app.py): `app.py`

### Persistent State

- [Result cache](https://github.com/rpc1418/document-retrieval-system/blob/main/app.py): `app.py`
- [Document store](https://github.com/rpc1418/document-retrieval-system/blob/main/cache.db): `cache.db`

### Other components

- **Chat client**
- **Trusted news sites**

## Connections

- Chat client → Search endpoint: queries (evidence: `README.md`)
- Chat client → Health endpoint: checks health (evidence: `README.md`)
- Search endpoint → Request limiter: checks limit (evidence: `app.py`)
- Request limiter → Document store: reads/writes counts (evidence: `app.py`)
- Search endpoint → Result cache: checks cache (evidence: `app.py`)
- Result cache → Document store: reads/writes cache (evidence: `app.py`)
- Search endpoint → Ranked search: runs search (evidence: `app.py`)
- Ranked search → Document store: reads documents (evidence: `app.py`)
- Ranked search → TF-IDF scoring: scores matches (evidence: `app.py`)
- Search endpoint → Result cache: stores results (evidence: `app.py`)
- Standalone search → Document store: reads documents (evidence: `fetchCode.py`)
- Standalone search → TF-IDF scoring: scores matches (evidence: `fetchCode.py`)
- Scraper thread → Article fetching: runs scraper (evidence: `app.py`)
- Article fetching → Trusted news sites: fetches pages (evidence: `app.py`)
- Article fetching → Article parsing: parses responses (evidence: `app.py`)
- Article parsing → Title deduplication: checks titles (evidence: `app.py`)
- Title deduplication → Document store: checks existing (evidence: `app.py`)
- Article parsing → Article storage: passes articles (evidence: `app.py`)
- Article storage → Document store: inserts articles (evidence: `app.py`)
