# linkedinlearning/rag-models-from-scratch-with-open-source-3980304 architecture

> Architecture diagram of the GitHub repository linkedinlearning/rag-models-from-scratch-with-open-source-3980304, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304
- Repository: https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304

## Overview

##### Project type and purpose
- This is a hands-on instructional repository for a Retrieval-Augmented Generation (RAG) pipeline built from scratch using open-source tools.
- Its purpose is to guide learners through:
- Installing a local LLM (LLama/ollama)
- Creating a content corpus and extracting text
- Generating embeddings and storing them in a vector database (Postgres + pgvector)
- Building a RAG service that retrieves context and calls the LLM to answer queries

##### File structure
- Top-level scripts prefixed with `b_`, `c_`, `d_` correspond to successive course stages:
- b_01…b_05: model install, model file prep, helper functions, corpus generation, text extraction
- c_02…c_05: Postgres install, DB connection, populating vector DB, querying embeddings
- d_02…d_05: content preparation and running the final RAG pipeline
- `init-db/init.sql`: schema for Postgres vector store
- `.devcontainer`: Dockerfile and config for reproducible dev environment
- GitHub workflows and templates for CI/CD and contribution guidelines

##### Architectural insights from README
- Emphasis on modular pipeline steps (install → prepare data → embed → store → retrieve → generate)
- Use of open-source LLMs and vector DB for efficient local experimentation

a. Developer Environment
- Dev Container (Docker)
- Shell & Python Scripts
b. Data Ingestion & Processing
- Corpus files (raw data sources)
- `generate_corpus.py` & `extract_text.py`
c. Embedding & Vector Store
- `ollama_functions.py` (LLM wrapper for embeddings)
- Postgres + pgvector (initialized via init.sql & `c_02…c_05` scripts)
d. Retrieval & Generation Service
- `run_rag.py`
- Querying the vector DB for similar contexts
- Passing context + prompt to local LLM for answer
e. Supporting Infrastructure
- GitHub Actions (CI/CD)
- Configuration files (.devcontainer, Dockerfile, workflows)

##### Architectural patterns & design principles
- Pipeline/ETL pattern: clearly separated stages for data preparation, embedding, storage, retrieval, and generation
- Modular scripting: each script handles one responsibility, enabling incremental learning and debugging
- Infrastructure as code: using Docker and SQL schema for reproducibility

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Developer Environment
    subgraph "Developer Environment"
        Dockerfile["Dockerfile"]:::infra
        DevcontainerConfig["devcontainer.json"]:::infra
    end

    %% Data Ingestion & Processing
    subgraph "Data Ingestion & Processing"
        RawData["Raw Data Sources"]:::compute
        GenerateCorpus["generate_corpus.py"]:::compute
        ExtractText["extract_text.py"]:::compute
    end

    %% Embedding & Vector Store
    subgraph "Embedding & Vector Store"
        OllamaFuncs["ollama_functions.py"]:::compute
        PostgresInit["init.sql"]:::storage
        PostgresSetup["postgres_installation.sh"]:::storage
        DBConnect["database_connect_embeddings.py"]:::compute
        PopulateDB["populate_vector_db.py"]:::compute
        PullDB["pull_db_content.py"]:::compute
    end

    %% Retrieval & Generation Service
    subgraph "Retrieval & Generation Service"
        PrepareContent["prepare_content.py"]:::compute
        RunRAG["run_rag.py"]:::compute
    end

    %% Supporting Infrastructure
    subgraph "Supporting Infrastructure"
        CIWorkflow["main.yml"]:::infra
        Codeowners["CODEOWNERS"]:::infra
        IssueTemplate["ISSUE_TEMPLATE.md"]:::infra
        PRTemplate["PULL_REQUEST_TEMPLATE.md"]:::infra
        Contributing["CONTRIBUTING.md"]:::infra
    end

    %% Data Flow
    RawData -->|ingest| GenerateCorpus -->|extract| ExtractText -->|embed| OllamaFuncs -->|insert| DBConnect -->|populate| PopulateDB -->|stored in| PostgresInit
    PostgresSetup --> PostgresInit
    RunRAG -->|query| PullDB -->|context| RunRAG
    RunRAG -->|prepare| PrepareContent -->|calls| OllamaFuncs

    %% Environment Provisioning
    Dockerfile --> GenerateCorpus
    DevcontainerConfig --> RunRAG

    %% CI/CD and Contribution links
    CIWorkflow --> GenerateCorpus
    Codeowners --> CIWorkflow
    IssueTemplate --> CIWorkflow
    PRTemplate --> CIWorkflow
    Contributing --> CIWorkflow

    %% Click Events
    click Dockerfile "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/tree/main/.devcontainer/Dockerfile"
    click DevcontainerConfig "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/blob/main/.devcontainer/devcontainer.json"
    click CIWorkflow "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/blob/main/.github/workflows/main.yml"
    click Codeowners "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/tree/main/.github/CODEOWNERS"
    click IssueTemplate "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/blob/main/.github/ISSUE_TEMPLATE.md"
    click PRTemplate "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/blob/main/.github/PULL_REQUEST_TEMPLATE.md"
    click Contributing "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/blob/main/CONTRIBUTING.md"
    click GenerateCorpus "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/blob/main/b_04_generate_corpus.py"
    click ExtractText "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/blob/main/b_05_extract_text.py"
    click OllamaFuncs "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/blob/main/b_03_ollama_functions.py"
    click PostgresInit "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/blob/main/init-db/init.sql"
    click PostgresSetup "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/blob/main/c_02_postgres_installation.sh"
    click DBConnect "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/blob/main/c_03_database_connect_embeddings.py"
    click PopulateDB "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/blob/main/c_04_populate_vector_db.py"
    click PullDB "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/blob/main/c_05_pull_db_content.py"
    click PrepareContent "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/blob/main/d_02_prepare_content.py"
    click RunRAG "https://github.com/linkedinlearning/rag-models-from-scratch-with-open-source-3980304/blob/main/d_05_run_rag.py"

    %% Styles
    classDef infra fill:#f39c12,stroke:#e67e22,color:#ffffff
    classDef compute fill:#2ecc71,stroke:#27ae60,color:#ffffff
    classDef storage fill:#3498db,stroke:#2980b9,color:#ffffff
```
