# a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry architecture

> Architecture diagram of the GitHub repository a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry
- Repository: https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry

## Overview

PRAG is a local Arabic-poetry retrieval-augmented question-answering system. Users submit questions through the Gradio interface; the API retrieves relevant indexed poem passages, generates an answer with either Ollama or AI21, and returns citations and timing. A separate corpus-preparation workflow scrapes, normalizes, chunks, and indexes poems. Relationships among sampled runtime modules are supported by source excerpts; preparation-stage links are documented in the README and shown in the sampled scripts.

Main entry points: [`api/main.py`](https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/api/main.py) (FastAPI endpoints), [`embeddings/embed_and_index.py`](https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/embeddings/embed_and_index.py) (Embedding and indexing), [`rag/retriever.py`](https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/rag/retriever.py) (Semantic retrieval), [`rag/generator.py`](https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/rag/generator.py) (Answer generation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["User interface"]
  node_ui["Gradio interface<br/>[app.py]"]
end

subgraph group_runtime["Question answering"]
  node_api["FastAPI endpoints<br/>[main.py]"]
  node_retriever["Semantic retrieval<br/>[retriever.py]"]
  node_generator["Answer generation<br/>[generator.py]"]
end

subgraph group_corpus["Corpus preparation"]
  node_scraper["Poem scraper<br/>[scrape_aldiwan.py]"]
  node_preprocessor["Arabic cleaning and chunking<br/>[clean_and_chunk.py]"]
  node_embedder["Embedding and indexing<br/>[embed_and_index.py]"]
end

subgraph group_storage["Poem data and index"]
  node_raw["Raw poem collection<br/>[poems_raw.json]"]
  node_chunks["Poem chunks<br/>[poems_chunks.json]"]
  node_faiss[("FAISS vector index<br/>[faiss.index]")]
  node_metadata["Chunk metadata<br/>[metadata.json]"]
end

node_user(("Poetry questioner"))
node_ollama["Ollama model"]
node_ai21["AI21 Jamba"]
node_aldiwan["AlDiwan website"]

node_user -->|"asks through"| node_ui
node_ui -->|"calls endpoints"| node_api
node_api -->|"retrieves chunks"| node_retriever
node_api -->|"generates answer"| node_generator
node_api -->|"returns response"| node_ui
node_retriever -->|"searches index"| node_faiss
node_retriever -->|"loads chunk metadata"| node_metadata
node_generator -->|"calls model"| node_ollama
node_generator -->|"calls model"| node_ai21
node_scraper -->|"scrapes poems"| node_aldiwan
node_scraper -->|"writes poems"| node_raw
node_preprocessor -->|"reads poems"| node_raw
node_preprocessor -->|"writes chunks"| node_chunks
node_embedder -->|"reads chunks"| node_chunks
node_embedder -->|"writes vectors"| node_faiss
node_embedder -->|"writes metadata"| node_metadata
node_api -->|"reads for stats"| node_raw
node_api -->|"reads for stats"| node_chunks
node_api -->|"reads index stats"| node_faiss

click node_ui "https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/ui/app.py"
click node_api "https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/api/main.py"
click node_retriever "https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/rag/retriever.py"
click node_generator "https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/rag/generator.py"
click node_scraper "https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/scraper/scrape_aldiwan.py"
click node_preprocessor "https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/preprocessing/clean_and_chunk.py"
click node_embedder "https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/embeddings/embed_and_index.py"
click node_raw "https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/data/poems_raw.json"
click node_chunks "https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/data/poems_chunks.json"
click node_faiss "https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/index/faiss.index"
click node_metadata "https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/index/metadata.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui toneBlue
class node_api,node_retriever,node_generator toneAmber
class node_scraper,node_preprocessor,node_embedder toneMint
class node_raw,node_chunks,node_faiss,node_metadata toneRose
class node_user,node_ollama,node_ai21,node_aldiwan toneIndigo
```

## Components

### User interface

- [Gradio interface](https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/ui/app.py): `ui/app.py`

### Question answering

- [FastAPI endpoints](https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/api/main.py): `api/main.py`
- [Semantic retrieval](https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/rag/retriever.py): `rag/retriever.py`
- [Answer generation](https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/rag/generator.py): `rag/generator.py`

### Corpus preparation

- [Poem scraper](https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/scraper/scrape_aldiwan.py): `scraper/scrape_aldiwan.py`
- [Arabic cleaning and chunking](https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/preprocessing/clean_and_chunk.py): `preprocessing/clean_and_chunk.py`
- [Embedding and indexing](https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/embeddings/embed_and_index.py): `embeddings/embed_and_index.py`

### Poem data and index

- [Raw poem collection](https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/data/poems_raw.json): `data/poems_raw.json`
- [Poem chunks](https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/data/poems_chunks.json): `data/poems_chunks.json`
- [FAISS vector index](https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/index/faiss.index): `index/faiss.index`
- [Chunk metadata](https://github.com/a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry/blob/main/index/metadata.json): `index/metadata.json`

### Other components

- **Poetry questioner**
- **Ollama model**
- **AI21 Jamba**
- **AlDiwan website**

## Connections

- Poetry questioner → Gradio interface: asks through (evidence: `README.md`)
- Gradio interface → FastAPI endpoints: calls endpoints (evidence: `ui/app.py`)
- FastAPI endpoints → Semantic retrieval: retrieves chunks (evidence: `api/main.py`)
- FastAPI endpoints → Answer generation: generates answer (evidence: `api/main.py`)
- FastAPI endpoints → Gradio interface: returns response (evidence: `ui/app.py`)
- Semantic retrieval → FAISS vector index: searches index (evidence: `rag/retriever.py`)
- Semantic retrieval → Chunk metadata: loads chunk metadata (evidence: `rag/retriever.py`)
- Answer generation → Ollama model: calls model (evidence: `rag/generator.py`)
- Answer generation → AI21 Jamba: calls model (evidence: `rag/generator.py`)
- Poem scraper → AlDiwan website: scrapes poems (evidence: `scraper/scrape_aldiwan.py`)
- Poem scraper → Raw poem collection: writes poems (evidence: `scraper/scrape_aldiwan.py`)
- Arabic cleaning and chunking → Raw poem collection: reads poems (evidence: `preprocessing/clean_and_chunk.py`)
- Arabic cleaning and chunking → Poem chunks: writes chunks (evidence: `preprocessing/clean_and_chunk.py`)
- Embedding and indexing → Poem chunks: reads chunks (evidence: `embeddings/embed_and_index.py`)
- Embedding and indexing → FAISS vector index: writes vectors (evidence: `embeddings/embed_and_index.py`)
- Embedding and indexing → Chunk metadata: writes metadata (evidence: `embeddings/embed_and_index.py`)
- FastAPI endpoints → Raw poem collection: reads for stats (evidence: `api/main.py`)
- FastAPI endpoints → Poem chunks: reads for stats (evidence: `api/main.py`)
- FastAPI endpoints → FAISS vector index: reads index stats (evidence: `api/main.py`)
