How henryhengluo/retrieval-augmented-generation-intro-project works
Project Type and Purpose
• This is an educational/demo pipeline project built around Jupyter notebooks that showcase Retrieval-Augmented Generation (RAG) using LlamaIndex and OpenAI’s API.
• Its goal is to load documents, build or query an index, retrieve relevant passages, and generate augmented text.
File Structure
• Top-level folders:
• code/ – three notebooks (basic pipeline, windowed retrieval, auto-merging retrieval)
• common/ – environment variable file (OpenAI key) and utility functions
• data/ – sample document (Henry.txt), user-provided docs go here
• files/ – project PDF and README for context
• python_env/ – YAML file for environment reproducibility
• Notice separation of concerns: data ingestion (data/), configuration (common/), core logic (code/), docs (files/), environment spec (python_env/).
README Insights
• Dependencies: Python, Jupyter, LlamaIndex, OpenAI models
• Workflow steps:
a. Environment setup (NLP.yml)
b. API key injection (common/openAI.env)
c. Read reference PDF
d. Run notebooks in sequence to see the RAG pipeline evolve
Main Components
a. User Interface: Jupyter Notebook server and client
b. Configuration: .env loader and YAML environment
c. Data Store: local text files in data/
d. Indexing Module: LlamaIndex library functions (via common/utils.py and notebook cells)
e. Retrieval Engine: vector store or in-memory index
f. Generation Service: OpenAI API (external HTTP service)
g. Notebook Logic: three stages—basic pipeline, sentence-window retrieval, auto-merge retrieval
Relationships and Data Flows
1) User writes/edits code in Jupyter →
2) Notebook loads data from data/ →
3) Notebook calls LlamaIndex to build/query index →
4) Retrieval results feed into OpenAI API requests →
5) API returns generated text →
6) Notebook displays output.
• Utilities in common/utils.py mediate environment loading and API calls.
Architectural Patterns and Principles
• Modular Pipeline: three successive experiments, each reusing the core retrieval + generation pattern.
• Separation of Concerns:
• Data ingest vs. indexing vs. generation
• Config management vs. business logic
• External Integration: treat OpenAI as an external microservice with clear HTTP boundaries.
Details to Capture
• Configuration load step (read .env, activate conda env)
• Index creation vs. query path
• How “auto-merging” differs: show an internal merge sub-component in the retrieval engine
• Flow of PDF/documentation reference (files/Intro*.pdf) as “User Reference” off to the side