How yusuf-br/pararag works
ParaRAG ingests PDFs into page-attributed chunks, loads a shared-rarity TF-IDF index across TCP shard processes, then answers questions by gathering shard results and optionally asking Groq to generate a cited response. The CLI also exposes shard kill/revive controls. The README documents filtering, damaged-PDF handling, and degraded-mode behavior; source excerpts show the core implementation. Ingestion and answer formatting share their respective modules, so those concerns are represented as components at the same paths.
Main entry points: main.py Interactive CLI · master.py Cluster coordinator · shard.py Shard server · generate.py Cited answer generation
Components 15
PDF ingestion
- Parallel extraction
ingest.py - Page chunks
ingest.py
Search cluster
- Cluster coordinator
master.py - Scatter-gather
master.py - Global word rarity
shard.py - Shard server
shard.py - TF-IDF index
shard.py - TCP message framing
net.py - Ranked passages
master.py
Answer generation
- Cited answer generation
generate.py
Other components
- PDF collection
- Interactive CLI
main.py - User question
- Groq LLM API
- Cited answer
Connections 22
- PDF collection → Interactive CLI: selects files
main.py - Interactive CLI → Parallel extraction: invokes ingestion
main.py - Parallel extraction → Page chunks: extracts and chunks
ingest.py - Interactive CLI → Cluster coordinator: starts and loads
main.py - Cluster coordinator → Global word rarity: computes frequencies
master.py - Cluster coordinator → Scatter-gather: dispatches shard messages
master.py - Scatter-gather → TCP message framing: sends and receives
master.py - TCP message framing → Shard server: transfers commands
shard.py - Cluster coordinator → Shard server: starts processes
master.py - Shard server → TCP message framing: reads and replies
shard.py - Shard server → TF-IDF index: loads and searches
shard.py - Global word rarity → TF-IDF index: provides global frequencies
master.py - Page chunks → Cluster coordinator: loads corpus
main.py - User question → Interactive CLI: enters query
main.py - Interactive CLI → Cluster coordinator: submits search
main.py - Scatter-gather → Shard server: queries concurrently
master.py - Cluster coordinator → Ranked passages: merges top results
master.py - Interactive CLI → Cited answer generation: passes question and hits
main.py - Cited answer generation → Groq LLM API: requests completion
generate.py - Cited answer generation → Cited answer: returns generated text
generate.py - Ranked passages → Interactive CLI: prints passages
main.py - Interactive CLI → Cluster coordinator: kills or revives shards
main.py