How llewis257/open-source-retrieval-augmented-q-a-pipeline works
This is an end-to-end RAG application: scheduled Airflow workflows extract and chunk local PDFs or web pages, then embed and store the text; a FastAPI query endpoint embeds questions, retrieves similar chunks, and asks Ollama to answer from that context. The graph includes the README’s monitoring capability, though its runtime wiring is documented rather than sampled. Source excerpts do not establish how the two DAGs are sequenced, so no dependency between them is shown.
Main entry points: main.py Query API · ingestion_dag.py Ingestion workflow · vectorization_dag.py Vectorization workflow · database.py Database access
Components 18
Ingestion
- Raw documents
data/raw - Ingestion workflow
airflow/dags/ingestion_dag.py - PDF extractor
airflow/plugins/scripts/pdf_extractor.py - Web extractor
airflow/plugins/scripts/web_extractor.py - Text chunker
airflow/plugins/scripts/text_processor.py - Processed chunks
data
Vector indexing
- Vectorization workflow
airflow/dags/vectorization_dag.py - Embedding model
airflow/plugins/scripts/vectorizer.py
Query service
- Query API
api/main.py - Ollama LLM
- Answer and context
Storage and monitoring
- Database access
airflow/plugins/scripts/database.py - PostgreSQL vectors
- Prometheus metrics
monitoring/prometheus.yml - Grafana dashboard
monitoring/grafana/dashboards/fastapi_dashboard.json
Other components
- Question asker
- Airflow scheduler
- Web pages
Connections 21
- Airflow scheduler → Ingestion workflow: schedules
- Airflow scheduler → Vectorization workflow: schedules
- Ingestion workflow → Raw documents: reads
- Ingestion workflow → PDF extractor: calls
- Ingestion workflow → Web extractor: calls
- Web extractor → Web pages: fetches
- Ingestion workflow → Text chunker: calls
- Ingestion workflow → Processed chunks: writes chunks
- Vectorization workflow → Processed chunks: reads chunks
- Vectorization workflow → Embedding model: calls
- Vectorization workflow → Database access: calls
- Database access → PostgreSQL vectors: creates and writes
- Question asker → Query API: asks
- Query API → Embedding model: embeds question
- Query API → Database access: opens connection
- Query API → PostgreSQL vectors: searches chunks
- Query API → Ollama LLM: sends grounded prompt
- Query API → Answer and context: returns
- Answer and context → Question asker: delivers
- Prometheus metrics → Query API: scrapes metrics
- Grafana dashboard → Prometheus metrics: visualizes