# nebeyoumusie/rag-pipeline architecture

> Architecture diagram of the GitHub repository nebeyoumusie/rag-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nebeyoumusie/rag-pipeline
- Repository: https://github.com/nebeyoumusie/rag-pipeline

## Overview

##### Project type and purpose
- This is a demonstration/research‐style Python project implementing a Retrieval-Augmented Generation (RAG) pipeline inside a Jupyter notebook.
- Purpose: augment a large language model’s (LLM) knowledge with private or post-training data (PDFs, text files) by retrieving relevant context at query time.

##### File structure
- Top level:
- README.md – project overview and instructions
- requirements.txt – Python dependencies (langchain, chromadb, bs4, pypdf, etc.)
- files/ – sample documents (attention.pdf, speech.txt)
- image/ – visual depiction of the RAG workflow (rag.png)
- notebook/ – rag_pipeline.ipynb containing the end-to-end demo
- No separate “frontend” or “backend” directories: the notebook is both driver and orchestrator.
- Key configurations: Python environment via requirements.txt.

##### README for additional insights
- Libraries: langchain for chains, chromadb for vector store, pypdf & bs4 for parsing.
- Workflow described: ingest documents → embed → index → retrieve → LLM generate.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Documentation and Config
    README["README.md\n(Project Overview & Instructions)"]:::external
    Requirements["requirements.txt\n(Python Dependencies)"]:::external
    VisualDiagram["Visual Workflow Diagram\n(rag.png)"]:::external

    %% Data Sources
    DocumentSources["Document Sources\n(files/)"]:::ingestion
    PDFDoc["Sample PDF Document\n(attention.pdf)"]:::ingestion
    TextDoc["Sample Text Document\n(speech.txt)"]:::ingestion

    %% Notebook Orchestrator
    subgraph "Notebook Orchestrator" 
        direction TB
        Notebook["rag_pipeline.ipynb\n(Orchestrator)"]:::notebook
        Loader["Document Loader"]:::ingestion
        Splitter["Text Splitter"]:::ingestion
        Embedder["Embedding Model\n(OpenAI via LangChain)"]:::ingestion
        Store["Vector Store\n(ChromaDB)"]:::storage
        Retriever["Retrieval Module"]:::retrieval
        LLMModule["LLM Generation Module\n(OpenAI via LangChain)"]:::retrieval
    end

    %% External API
    ExternalAPI["OpenAI API\n(External Service)"]:::external

    %% Data Flow
    DocumentSources --> PDFDoc
    DocumentSources --> TextDoc
    DocumentSources --> Loader
    PDFDoc --> Loader
    TextDoc --> Loader
    Loader --> Splitter
    Splitter --> Embedder
    Embedder --> Store
    %% Retrieval flow
    UserQuery["User Query"]:::external --> Retriever
    Store --> Retriever
    Retriever --> LLMModule
    LLMModule --> Response["Augmented Answer"]:::external
    LLMModule --> ExternalAPI
    ExternalAPI --> LLMModule

    %% Notebook orchestration links
    Notebook --> Loader
    Notebook --> Splitter
    Notebook --> Embedder
    Notebook --> Store
    Notebook --> Retriever
    Notebook --> LLMModule

    %% Click Events
    click DocumentSources "https://github.com/nebeyoumusie/rag-pipeline/tree/main/files/"
    click PDFDoc "https://github.com/nebeyoumusie/rag-pipeline/blob/main/files/attention.pdf"
    click TextDoc "https://github.com/nebeyoumusie/rag-pipeline/blob/main/files/speech.txt"
    click README "https://github.com/nebeyoumusie/rag-pipeline/blob/main/README.md"
    click Requirements "https://github.com/nebeyoumusie/rag-pipeline/blob/main/requirements.txt"
    click VisualDiagram "https://github.com/nebeyoumusie/rag-pipeline/blob/main/image/rag.png"
    click Notebook "https://github.com/nebeyoumusie/rag-pipeline/blob/main/notebook/rag_pipeline.ipynb"
    click Loader "https://github.com/nebeyoumusie/rag-pipeline/blob/main/notebook/rag_pipeline.ipynb"
    click Splitter "https://github.com/nebeyoumusie/rag-pipeline/blob/main/notebook/rag_pipeline.ipynb"
    click Embedder "https://github.com/nebeyoumusie/rag-pipeline/blob/main/notebook/rag_pipeline.ipynb"
    click Store "https://github.com/nebeyoumusie/rag-pipeline/blob/main/notebook/rag_pipeline.ipynb"
    click Retriever "https://github.com/nebeyoumusie/rag-pipeline/blob/main/notebook/rag_pipeline.ipynb"
    click LLMModule "https://github.com/nebeyoumusie/rag-pipeline/blob/main/notebook/rag_pipeline.ipynb"
    click ExternalAPI "https://github.com/nebeyoumusie/rag-pipeline/blob/main/requirements.txt"

    %% Styles
    classDef ingestion fill:#cce5ff,stroke:#004085,color:#004085;
    classDef storage fill:#fff3cd,stroke:#856404,color:#856404;
    classDef retrieval fill:#d4edda,stroke:#155724,color:#155724;
    classDef notebook fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
    classDef external fill:#f8f9fa,stroke:#6c757d,color:#6c757d;
```
