# henryhengluo/retrieval-augmented-generation-intro-project architecture

> Architecture diagram of the GitHub repository henryhengluo/retrieval-augmented-generation-intro-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/henryhengluo/retrieval-augmented-generation-intro-project
- Repository: https://github.com/henryhengluo/retrieval-augmented-generation-intro-project

## Overview

##### Project Type and Purpose
- This is an educational/demo pipeline project built around Jupyter notebooks that showcase Retrieval-Augmented Generation (RAG) using LlamaIndex and OpenAI’s API.
- Its goal is to load documents, build or query an index, retrieve relevant passages, and generate augmented text.

##### File Structure
- Top-level folders:
- code/ – three notebooks (basic pipeline, windowed retrieval, auto-merging retrieval)
- common/ – environment variable file (OpenAI key) and utility functions
- data/ – sample document (Henry.txt), user-provided docs go here
- files/ – project PDF and README for context
- python_env/ – YAML file for environment reproducibility
- Notice separation of concerns: data ingestion (data/), configuration (common/), core logic (code/), docs (files/), environment spec (python_env/).

##### README Insights
- Dependencies: Python, Jupyter, LlamaIndex, OpenAI models
- Workflow steps:
a. Environment setup (NLP.yml)
b. API key injection (common/openAI.env)
c. Read reference PDF
d. Run notebooks in sequence to see the RAG pipeline evolve

##### Main Components
a. **User Interface**: Jupyter Notebook server and client
b. **Configuration**: `.env` loader and YAML environment
c. **Data Store**: local text files in `data/`
d. **Indexing Module**: LlamaIndex library functions (via common/utils.py and notebook cells)
e. **Retrieval Engine**: vector store or in-memory index
f. **Generation Service**: OpenAI API (external HTTP service)
g. **Notebook Logic**: three stages—basic pipeline, sentence-window retrieval, auto-merge retrieval

##### Relationships and Data Flows
1) User writes/edits code in Jupyter →
2) Notebook loads data from `data/` →
3) Notebook calls LlamaIndex to build/query index →
4) Retrieval results feed into OpenAI API requests →
5) API returns generated text →
6) Notebook displays output.
- Utilities in `common/utils.py` mediate environment loading and API calls.

##### Architectural Patterns and Principles
- **Modular Pipeline**: three successive experiments, each reusing the core retrieval + generation pattern.
- **Separation of Concerns**:
- Data ingest vs. indexing vs. generation
- Config management vs. business logic
- **External Integration**: treat OpenAI as an external microservice with clear HTTP boundaries.

##### Details to Capture
- Configuration load step (read `.env`, activate conda env)
- Index creation vs. query path
- How “auto-merging” differs: show an internal merge sub-component in the retrieval engine
- Flow of PDF/documentation reference (files/Intro*.pdf) as “User Reference” off to the side

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Configuration"
        Env[".env"]:::config
        YAML["NLP.yml"]:::config
    end

    subgraph "Notebook Experiments"
        NB1["1_Basic_RAG_Pipeline.ipynb"]:::notebook
        NB2["2_Sentence_window_retrieval.ipynb"]:::notebook
        NB3["3_Auto-merging_Retrieval.ipynb"]:::notebook
    end

    Utils["utils.py"]:::local
    subgraph "Data Store"
        DataFolder["data/"]:::data
        HenryTxt["Henry.txt"]:::data
    end

    subgraph "Documentation"
        PDF["Intro of Retrieval Augmented Generation (RAG) and application demos_Henry.pdf"]:::docs
        Readme["readme.md"]:::docs
    end

    OpenAI["OpenAI API"]:::external

    %% Data flow
    DataFolder --> HenryTxt
    DataFolder --> NB1
    DataFolder --> NB2
    DataFolder --> NB3

    %% Notebook dependencies
    NB1 --> Utils
    NB2 --> Utils
    NB3 --> Utils

    %% Config and environment
    Env --> Utils
    YAML --> NB1
    YAML --> NB2
    YAML --> NB3

    %% Retrieval & generation
    Utils --> OpenAI
    OpenAI --> NB1
    OpenAI --> NB2
    OpenAI --> NB3

    %% Documentation reference
    PDF -.-> NB1
    Readme -.-> NB1

    %% Click events
    click NB1 "https://github.com/henryhengluo/retrieval-augmented-generation-intro-project/blob/main/code/1_Basic_RAG_Pipeline.ipynb"
    click NB2 "https://github.com/henryhengluo/retrieval-augmented-generation-intro-project/blob/main/code/2_Sentence_window_retrieval.ipynb"
    click NB3 "https://github.com/henryhengluo/retrieval-augmented-generation-intro-project/blob/main/code/3_Auto-merging_Retrieval.ipynb"
    click Env "https://github.com/henryhengluo/retrieval-augmented-generation-intro-project/blob/main/common/openAI.env"
    click YAML "https://github.com/henryhengluo/retrieval-augmented-generation-intro-project/blob/main/python_env/NLP.yml"
    click DataFolder "https://github.com/henryhengluo/retrieval-augmented-generation-intro-project/tree/main/data/"
    click HenryTxt "https://github.com/henryhengluo/retrieval-augmented-generation-intro-project/blob/main/data/Henry.txt"
    click Utils "https://github.com/henryhengluo/retrieval-augmented-generation-intro-project/blob/main/common/utils.py"
    click PDF "https://github.com/henryhengluo/retrieval-augmented-generation-intro-project/blob/main/files/Intro of Retrieval Augmented Generation (RAG) and application demos_Henry.pdf"
    click Readme "https://github.com/henryhengluo/retrieval-augmented-generation-intro-project/blob/main/files/readme.md"

    classDef notebook fill:#D0E6FF,stroke:#0050B3
    classDef local fill:#E8F4FF,stroke:#1E90FF
    classDef data fill:#E6FFE6,stroke:#2E8B57,stroke-width:2px
    classDef external fill:#FFE6D6,stroke:#FF8C00
    classDef config fill:#F0F0F0,stroke:#CCCCCC,stroke-dasharray: 5 5
    classDef docs fill:#FFF5E6,stroke:#DAA520
```
