How datastax/graph-rag-example works
Project type and purpose
• This is a Python-based RAG (Retrieval-Augmented Generation) demo app that:
• Ingests structured content (movie pages) from TMDB via a loader script.
• Stores embeddings and graph relationships in a Cassandra-backed graph vector store (DataStax Astra DB).
• Exposes a Dash UI for real-time comparison of vector similarity vs. graph traversal queries, powered by an OpenAI LLM.
File structure
• Top-level scripts:
• load_data.py – ingestion pipeline (scraping, processing, upserting into vector/graph store)
• app.py – Dash application entry point
• search_executor.py – encapsulates search logic (similarity & graph traversal)
• Utilities in util/:
• config.py – environment variable parsing (Astra DB credentials, OpenAI key)
• scrub.py – text cleaning before embedding
• visualization.py – helpers for building UI graphs or charts
• assets/: static files (CSS, sample movies.json)
• requirements.txt & .env.example – dependencies and configuration boilerplate
README for additional insights
• Confirms tech stack: Python 3, Dash (Plotly), LangChain graph_vectorstores, CassandraGraphVectorStore (DataStax Astra), OpenAI API
• Usage flow:
1. load_data.py → pulls & preprocesses TMDB content
2. app.py → runs Dash UI for user queries, delegates to search_executor
a. Main components:
1. External Data Source (TMDB API)
2. Data Ingestion Layer (load_data.py)
3. Data Processing & Utilities (util/scrub.py, util/config.py)
4. Graph Vector Store (DataStax Astra Cassandra)
5. Search Layer (search_executor.py)
6. LLM Service (OpenAI API)
7. Presentation Layer (Dash UI in app.py)
b. Relationships & interactions:
• load_data.py → scrub.py (text cleaning) → CassandraGraphVectorStore (“upsert embeddings & graph edges”)
• User (browser) → Dash UI (HTTP) → search_executor
• search_executor → CassandraGraphVectorStore (two paths: similarity search & graph traversal)
• search_executor → OpenAI API (prompting & response generation)
• search_executor → Dash UI (returns results for rendering)
c. Architectural patterns & principles:
• Monolithic service divided into clear layers: ingestion, storage, search logic, presentation
• RAG pattern: combine vector similarity & structured graph traversal
• Single source of truth for config (.env → util/config.py)
• Separation of concerns: utilities (scrubbing, visualization) factored out
d. Key technologies/frameworks:
• Python 3 + LangChain graph_vectorstores
• DataStax Astra DB (CassandraGraphVectorStore)
• OpenAI LLM API
• Dash (Plotly) for UI
Detailed capture of essential elements
• Open ports or protocols where relevant (HTTP, REST API calls)
• If desired, annotate critical configuration parameters (Astra DB endpoint, OpenAI key) adjacent to util/config.py