How rpc1418/document-retrieval-system works
This is a Flask document-retrieval service: users query indexed news articles and receive similarity-ranked snippets. The main flow applies per-user request limits, checks SQLite cache, then searches stored documents with TF-IDF and cosine similarity. Separately, a background thread fetches articles from three news sites, parses them, and stores new documents. Health checks and a standalone search script are also represented. The runtime is concentrated in app.py; README-described behavior is included, but unsampled modules and deployment details are not expanded.
Main entry points: cache.db Document store · app.py Search endpoint · fetchCode.py Standalone search
Components 15
HTTP API
- Search endpoint
app.py - Health endpoint
app.py - Request limiter
app.py
Document Retrieval
- Ranked search
app.py - TF-IDF scoring
app.py - Standalone search
fetchCode.py
News Ingestion
- Scraper thread
app.py - Article fetching
app.py - Article parsing
app.py - Title deduplication
app.py - Article storage
app.py
Persistent State
- Result cache
app.py - Document store
cache.db
Other components
- Chat client
- Trusted news sites
Connections 19
- Chat client → Search endpoint: queries
README.md - Chat client → Health endpoint: checks health
README.md - Search endpoint → Request limiter: checks limit
app.py - Request limiter → Document store: reads/writes counts
app.py - Search endpoint → Result cache: checks cache
app.py - Result cache → Document store: reads/writes cache
app.py - Search endpoint → Ranked search: runs search
app.py - Ranked search → Document store: reads documents
app.py - Ranked search → TF-IDF scoring: scores matches
app.py - Search endpoint → Result cache: stores results
app.py - Standalone search → Document store: reads documents
fetchCode.py - Standalone search → TF-IDF scoring: scores matches
fetchCode.py - Scraper thread → Article fetching: runs scraper
app.py - Article fetching → Trusted news sites: fetches pages
app.py - Article fetching → Article parsing: parses responses
app.py - Article parsing → Title deduplication: checks titles
app.py - Title deduplication → Document store: checks existing
app.py - Article parsing → Article storage: passes articles
app.py - Article storage → Document store: inserts articles
app.py