# llewis257/open-source-retrieval-augmented-q-a-pipeline architecture

> Architecture diagram of the GitHub repository llewis257/open-source-retrieval-augmented-q-a-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/llewis257/open-source-retrieval-augmented-q-a-pipeline
- Repository: https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline

## Overview

This is an end-to-end RAG application: scheduled Airflow workflows extract and chunk local PDFs or web pages, then embed and store the text; a FastAPI query endpoint embeds questions, retrieves similar chunks, and asks Ollama to answer from that context. The graph includes the README’s monitoring capability, though its runtime wiring is documented rather than sampled. Source excerpts do not establish how the two DAGs are sequenced, so no dependency between them is shown.

Main entry points: [`api/main.py`](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/api/main.py) (Query API), [`airflow/dags/ingestion_dag.py`](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/dags/ingestion_dag.py) (Ingestion workflow), [`airflow/dags/vectorization_dag.py`](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/dags/vectorization_dag.py) (Vectorization workflow), [`airflow/plugins/scripts/database.py`](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/plugins/scripts/database.py) (Database access)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion"]
  node_raw["Raw documents"]
  node_ingestion_dag["Ingestion workflow<br/>[ingestion_dag.py]"]
  node_pdf["PDF extractor<br/>[pdf_extractor.py]"]
  node_web["Web extractor<br/>[web_extractor.py]"]
  node_processor["Text chunker<br/>[text_processor.py]"]
  node_processed["Processed chunks"]
end

subgraph group_index["Vector indexing"]
  node_vectorization_dag["Vectorization workflow"]
  node_vectorizer["Embedding model<br/>[vectorizer.py]"]
end

subgraph group_query["Query service"]
  node_api["Query API<br/>[main.py]"]
  node_ollama["Ollama LLM"]
  node_answer["Answer and context"]
end

subgraph group_platform["Storage and monitoring"]
  node_database_helper["Database access<br/>[database.py]"]
  node_postgres[("PostgreSQL vectors")]
  node_monitoring["Prometheus metrics<br/>[prometheus.yml]"]
  node_grafana["Grafana dashboard"]
end

node_user(("Question asker"))
node_airflow(("Airflow scheduler"))
node_websites(("Web pages"))

node_airflow -->|"schedules"| node_ingestion_dag
node_airflow -->|"schedules"| node_vectorization_dag
node_ingestion_dag -->|"reads"| node_raw
node_ingestion_dag -->|"calls"| node_pdf
node_ingestion_dag -->|"calls"| node_web
node_web -->|"fetches"| node_websites
node_ingestion_dag -->|"calls"| node_processor
node_ingestion_dag -->|"writes chunks"| node_processed
node_vectorization_dag -->|"reads chunks"| node_processed
node_vectorization_dag -->|"calls"| node_vectorizer
node_vectorization_dag -->|"calls"| node_database_helper
node_database_helper -->|"creates and writes"| node_postgres
node_user -->|"asks"| node_api
node_api -->|"embeds question"| node_vectorizer
node_api -->|"opens connection"| node_database_helper
node_api -->|"searches chunks"| node_postgres
node_api -->|"sends grounded prompt"| node_ollama
node_api -->|"returns"| node_answer
node_answer -->|"delivers"| node_user
node_monitoring -.->|"scrapes metrics"| node_api
node_grafana -.->|"visualizes"| node_monitoring

click node_raw "https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/tree/main/data/raw"
click node_ingestion_dag "https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/dags/ingestion_dag.py"
click node_pdf "https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/plugins/scripts/pdf_extractor.py"
click node_web "https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/plugins/scripts/web_extractor.py"
click node_processor "https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/plugins/scripts/text_processor.py"
click node_processed "https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/tree/main/data"
click node_vectorization_dag "https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/dags/vectorization_dag.py"
click node_vectorizer "https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/plugins/scripts/vectorizer.py"
click node_database_helper "https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/plugins/scripts/database.py"
click node_api "https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/api/main.py"
click node_monitoring "https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/monitoring/prometheus.yml"
click node_grafana "https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/monitoring/grafana/dashboards/fastapi_dashboard.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw,node_ingestion_dag,node_pdf,node_web,node_processor,node_processed toneBlue
class node_vectorization_dag,node_vectorizer toneAmber
class node_api,node_ollama,node_answer toneMint
class node_database_helper,node_postgres,node_monitoring,node_grafana,node_airflow toneRose
class node_user,node_websites toneIndigo
```

## Components

### Ingestion

- [Raw documents](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/tree/main/data/raw): `data/raw`
- [Ingestion workflow](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/dags/ingestion_dag.py): `airflow/dags/ingestion_dag.py`
- [PDF extractor](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/plugins/scripts/pdf_extractor.py): `airflow/plugins/scripts/pdf_extractor.py`
- [Web extractor](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/plugins/scripts/web_extractor.py): `airflow/plugins/scripts/web_extractor.py`
- [Text chunker](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/plugins/scripts/text_processor.py): `airflow/plugins/scripts/text_processor.py`
- [Processed chunks](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/tree/main/data): `data`

### Vector indexing

- [Vectorization workflow](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/dags/vectorization_dag.py): `airflow/dags/vectorization_dag.py`
- [Embedding model](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/plugins/scripts/vectorizer.py): `airflow/plugins/scripts/vectorizer.py`

### Query service

- [Query API](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/api/main.py): `api/main.py`
- **Ollama LLM**
- **Answer and context**

### Storage and monitoring

- [Database access](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/airflow/plugins/scripts/database.py): `airflow/plugins/scripts/database.py`
- **PostgreSQL vectors**
- [Prometheus metrics](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/monitoring/prometheus.yml): `monitoring/prometheus.yml`
- [Grafana dashboard](https://github.com/llewis257/open-source-retrieval-augmented-q-a-pipeline/blob/main/monitoring/grafana/dashboards/fastapi_dashboard.json): `monitoring/grafana/dashboards/fastapi_dashboard.json`

### Other components

- **Question asker**
- **Airflow scheduler**
- **Web pages**

## Connections

- Airflow scheduler → Ingestion workflow: schedules
- Airflow scheduler → Vectorization workflow: schedules
- Ingestion workflow → Raw documents: reads
- Ingestion workflow → PDF extractor: calls
- Ingestion workflow → Web extractor: calls
- Web extractor → Web pages: fetches
- Ingestion workflow → Text chunker: calls
- Ingestion workflow → Processed chunks: writes chunks
- Vectorization workflow → Processed chunks: reads chunks
- Vectorization workflow → Embedding model: calls
- Vectorization workflow → Database access: calls
- Database access → PostgreSQL vectors: creates and writes
- Question asker → Query API: asks
- Query API → Embedding model: embeds question
- Query API → Database access: opens connection
- Query API → PostgreSQL vectors: searches chunks
- Query API → Ollama LLM: sends grounded prompt
- Query API → Answer and context: returns
- Answer and context → Question asker: delivers
- Prometheus metrics → Query API: scrapes metrics
- Grafana dashboard → Prometheus metrics: visualizes