# yusuf-br/pararag architecture

> Architecture diagram of the GitHub repository yusuf-br/pararag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yusuf-br/pararag
- Repository: https://github.com/yusuf-br/pararag

## Overview

ParaRAG is a PDF question-answering system: it extracts page-aware passages in parallel, loads a sharded search cluster, gathers and merges shard results for each question, then optionally asks a Groq-hosted LLM to answer with citations. The CLI also exposes shard failure and recovery. The graph follows the sampled runtime calls and README-documented behavior; unsampled modules are not assumed to add further runtime stages.

Main entry points: [`master.py`](https://github.com/yusuf-br/pararag/blob/main/master.py) (Cluster manager), [`main.py`](https://github.com/yusuf-br/pararag/blob/main/main.py) (Interactive CLI), [`shard.py`](https://github.com/yusuf-br/pararag/blob/main/shard.py) (Shard server), [`generate.py`](https://github.com/yusuf-br/pararag/blob/main/generate.py) (Citation answer generator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["PDF ingestion"]
  node_ingest["Parallel extraction<br/>[ingest.py]"]
end

subgraph group_cluster["Sharded search"]
  node_cli["Interactive CLI<br/>[main.py]"]
  node_cluster_manager["Cluster manager<br/>[master.py]"]
  node_shard_server["Shard server<br/>[shard.py]"]
  node_local_index[("TF-IDF index<br/>[shard.py]")]
  node_protocol["TCP message framing<br/>[net.py]"]
end

subgraph group_answering["Answer generation"]
  node_answer_generator["Citation answer generator<br/>[generate.py]"]
end

node_pdfs["PDF collection"]
node_user(("CLI user"))
node_groq["Groq API"]

node_user -->|"asks questions"| node_cli
node_pdfs -->|"provides files"| node_cli
node_cli -->|"ingests PDFs"| node_ingest
node_ingest -->|"returns chunks"| node_cluster_manager
node_cli -->|"loads and queries"| node_cluster_manager
node_cluster_manager -->|"computes word rarity"| node_local_index
node_cluster_manager -->|"starts shard processes"| node_shard_server
node_cluster_manager -->|"sends shard requests"| node_protocol
node_shard_server -->|"receives shard requests"| node_protocol
node_shard_server -->|"loads and searches"| node_local_index
node_cluster_manager -->|"scatters load and search"| node_shard_server
node_shard_server -->|"returns top results"| node_cluster_manager
node_cluster_manager -->|"returns merged results"| node_cli
node_cli -->|"generates answer"| node_answer_generator
node_answer_generator -->|"requests completion"| node_groq
node_answer_generator -->|"returns cited answer"| node_cli
node_cli -->|"kills or revives shards"| node_cluster_manager

click node_cli "https://github.com/yusuf-br/pararag/blob/main/main.py"
click node_ingest "https://github.com/yusuf-br/pararag/blob/main/ingest.py"
click node_cluster_manager "https://github.com/yusuf-br/pararag/blob/main/master.py"
click node_shard_server "https://github.com/yusuf-br/pararag/blob/main/shard.py"
click node_local_index "https://github.com/yusuf-br/pararag/blob/main/shard.py"
click node_protocol "https://github.com/yusuf-br/pararag/blob/main/net.py"
click node_answer_generator "https://github.com/yusuf-br/pararag/blob/main/generate.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingest,node_user toneBlue
class node_cli,node_cluster_manager,node_shard_server,node_local_index,node_protocol toneAmber
class node_answer_generator,node_groq toneMint
class node_pdfs toneIndigo
```

## Components

### PDF ingestion

- [Parallel extraction](https://github.com/yusuf-br/pararag/blob/main/ingest.py): `ingest.py`

### Sharded search

- [Interactive CLI](https://github.com/yusuf-br/pararag/blob/main/main.py): `main.py`
- [Cluster manager](https://github.com/yusuf-br/pararag/blob/main/master.py): `master.py`
- [Shard server](https://github.com/yusuf-br/pararag/blob/main/shard.py): `shard.py`
- [TF-IDF index](https://github.com/yusuf-br/pararag/blob/main/shard.py): `shard.py`
- [TCP message framing](https://github.com/yusuf-br/pararag/blob/main/net.py): `net.py`

### Answer generation

- [Citation answer generator](https://github.com/yusuf-br/pararag/blob/main/generate.py): `generate.py`

### Other components

- **PDF collection**
- **CLI user**
- **Groq API**

## Connections

- CLI user → Interactive CLI: asks questions (evidence: `README.md`)
- PDF collection → Interactive CLI: provides files (evidence: `README.md`)
- Interactive CLI → Parallel extraction: ingests PDFs (evidence: `main.py`)
- Parallel extraction → Cluster manager: returns chunks (evidence: `main.py`)
- Interactive CLI → Cluster manager: loads and queries (evidence: `main.py`)
- Cluster manager → TF-IDF index: computes word rarity (evidence: `master.py`)
- Cluster manager → Shard server: starts shard processes (evidence: `master.py`)
- Cluster manager → TCP message framing: sends shard requests (evidence: `master.py`)
- Shard server → TCP message framing: receives shard requests (evidence: `shard.py`)
- Shard server → TF-IDF index: loads and searches (evidence: `shard.py`)
- Cluster manager → Shard server: scatters load and search (evidence: `master.py`)
- Shard server → Cluster manager: returns top results (evidence: `master.py`)
- Cluster manager → Interactive CLI: returns merged results (evidence: `main.py`)
- Interactive CLI → Citation answer generator: generates answer (evidence: `main.py`)
- Citation answer generator → Groq API: requests completion (evidence: `generate.py`)
- Citation answer generator → Interactive CLI: returns cited answer (evidence: `main.py`)
- Interactive CLI → Cluster manager: kills or revives shards (evidence: `main.py`)
