How a-elshamy1/prag-a-retrieval-augmented-system-for-understanding-arabic-poetry works
PRAG is a local Arabic-poetry retrieval-augmented question-answering system. Users submit questions through the Gradio interface; the API retrieves relevant indexed poem passages, generates an answer with either Ollama or AI21, and returns citations and timing. A separate corpus-preparation workflow scrapes, normalizes, chunks, and indexes poems. Relationships among sampled runtime modules are supported by source excerpts; preparation-stage links are documented in the README and shown in the sampled scripts.
Main entry points: main.py FastAPI endpoints · embed_and_index.py Embedding and indexing · retriever.py Semantic retrieval · generator.py Answer generation
Components 15
User interface
- Gradio interface
ui/app.py
Question answering
- FastAPI endpoints
api/main.py - Semantic retrieval
rag/retriever.py - Answer generation
rag/generator.py
Corpus preparation
- Poem scraper
scraper/scrape_aldiwan.py - Arabic cleaning and chunking
preprocessing/clean_and_chunk.py - Embedding and indexing
embeddings/embed_and_index.py
Poem data and index
- Raw poem collection
data/poems_raw.json - Poem chunks
data/poems_chunks.json - FAISS vector index
index/faiss.index - Chunk metadata
index/metadata.json
Other components
- Poetry questioner
- Ollama model
- AI21 Jamba
- AlDiwan website
Connections 19
- Poetry questioner → Gradio interface: asks through
README.md - Gradio interface → FastAPI endpoints: calls endpoints
ui/app.py - FastAPI endpoints → Semantic retrieval: retrieves chunks
api/main.py - FastAPI endpoints → Answer generation: generates answer
api/main.py - FastAPI endpoints → Gradio interface: returns response
ui/app.py - Semantic retrieval → FAISS vector index: searches index
rag/retriever.py - Semantic retrieval → Chunk metadata: loads chunk metadata
rag/retriever.py - Answer generation → Ollama model: calls model
rag/generator.py - Answer generation → AI21 Jamba: calls model
rag/generator.py - Poem scraper → AlDiwan website: scrapes poems
scraper/scrape_aldiwan.py - Poem scraper → Raw poem collection: writes poems
scraper/scrape_aldiwan.py - Arabic cleaning and chunking → Raw poem collection: reads poems
preprocessing/clean_and_chunk.py - Arabic cleaning and chunking → Poem chunks: writes chunks
preprocessing/clean_and_chunk.py - Embedding and indexing → Poem chunks: reads chunks
embeddings/embed_and_index.py - Embedding and indexing → FAISS vector index: writes vectors
embeddings/embed_and_index.py - Embedding and indexing → Chunk metadata: writes metadata
embeddings/embed_and_index.py - FastAPI endpoints → Raw poem collection: reads for stats
api/main.py - FastAPI endpoints → Poem chunks: reads for stats
api/main.py - FastAPI endpoints → FAISS vector index: reads index stats
api/main.py