# 12parker/longcontextrag architecture

> Architecture diagram of the GitHub repository 12parker/longcontextrag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/12parker/longcontextrag
- Repository: https://github.com/12parker/longcontextrag

## Overview

##### Project type and purpose
- This is a research framework for “Retrieval Augmented Generation” (RAG) optimized for very long contexts (up to 128k+ tokens). It combines an external vector store (ChromaDB) with large language models (via OpenAI API) and provides hybrid attention modules, evaluation pipelines, training scripts, and example clients.

##### File structure
- Top-level modules:
- core/ – core RAG system (config, indexing, prompt templates, long-context handling)
- hybrid/ – hybrid attention RAG extension (neural retriever, attention mechanisms, integration)
- VectorDB/ – vector database builder for BookCorpus (build_db.py)
- evaluation/ – evaluation framework (quick and full evaluations)
- examples/ – demo and interactive scripts
- training/ – scripts to train neural retrievers and hybrid RAG
- testing/ – test harnesses for BookCorpus integration and hybrid tests
- utils/ – environment setup, test setup, token usage analysis
- research/ – notebooks and experimentation tools
- Deployment/config:
- activate.sh, env.template, requirements.txt
- main.py – entry point for ad hoc runs
- Data directories (persisted outputs):
- vector_store_eval/, vector_store_long/ – serialized ChromaDB stores

##### README for additional insights
- External services: OpenAI API, ChromaDB
- Core features: chunking, TOP_K retrieval, configurable context window, multiple prompt templates, hybrid sliding/sparse attention strategies
- Usage patterns: build vector store → index documents → generate response with or without retrieval → evaluate performance

## Diagram (Mermaid)

```mermaid
graph TD
    %% Configuration & Environment Loader
    subgraph "Configuration & Environment Loader"
        Config1["core/config.py"]:::config
        Config2["utils/setup_env.py"]:::config
    end

    %% Data Ingestion & Vector Store Builder
    subgraph "Data Ingestion & Vector Store Builder"
        Ingest["VectorDB/build_db.py"]:::ingest
    end

    %% Vector Store
    subgraph "Vector Store"
        VS1["vector_store_eval/"]:::db
        VS2["vector_store_long/"]:::db
    end

    %% Core RAG Module
    subgraph "Core RAG Module"
        Index1["core/index.py"]:::core
        Index2["core/long_context_config.py"]:::core
        Index3["core/prompts.py"]:::core
    end

    %% Hybrid RAG Extension
    subgraph "Hybrid RAG Extension"
        H1["hybrid/hybrid_attention_rag.py"]:::ext
        H2["hybrid/hybrid_rag_integration.py"]:::ext
        H3["hybrid/working_hybrid_rag.py"]:::ext
    end

    %% LLM Service
    LLM["OpenAI API"]:::external

    %% Training Pipeline
    subgraph "Training Pipeline"
        T1["training/neural_retriever.py"]:::train
        T2["training/train_hybrid_rag.py"]:::train
    end

    %% Evaluation Framework
    subgraph "Evaluation Framework"
        E1["evaluation/quick_rag_evaluation.py"]:::eval
        E2["evaluation/rag_evaluation.py"]:::eval
    end

    %% Client Examples & Interactive Shell
    subgraph "Client Examples & Interactive Shell"
        C1["examples/compare_responses.py"]:::client
        C2["examples/examples.py"]:::client
        C3["examples/interactive_rag.py"]:::client
        C4["examples/long_context_testing.py"]:::client
        C5["examples/run_long_context_test.py"]:::client
        C6["main.py"]:::client
    end

    %% Utility Modules & Research Notebooks
    subgraph "Utility Modules & Research Notebooks"
        U1["utils/test_setup.py"]:::util
        U2["utils/token_usage_analysis.py"]:::util
        R1["research/research_notebook.py"]:::util
    end

    %% Relationships
    Config1 --> Ingest
    Config1 --> Index1
    Config1 --> H1
    Config1 --> T1
    Config1 --> E1
    Config1 --> C1
    Config1 --> U1

    Ingest --> VS1
    Ingest --> VS2

    C1 --> Index1
    C2 --> Index1
    C3 --> Index1
    C4 --> Index1
    C5 --> Index1
    C6 --> Index1

    Index1 -->|"vector query"| VS1
    Index1 -->|"vector query"| VS2
    VS1 -->|"retrieved docs"| Index1
    VS2 -->|"retrieved docs"| Index1

    Index1 -->|"prompt payload"| LLM
    LLM -->|"response"| Index1

    Index1 --> H1
    H1 -->|"augmented request"| LLM
    LLM --> H1
    H1 --> Index1

    T1 --> VS1
    T1 --> VS2
    T1 --> LLM
    T2 --> VS1
    T2 --> VS2
    T2 --> LLM

    E1 --> Index1
    E1 --> H1
    E1 --> VS1
    E1 --> VS2
    E2 --> Index1
    E2 --> H1
    E2 --> VS1
    E2 --> VS2

    U1 --> E1
    U1 --> E2
    U1 --> T1
    U1 --> T2
    U1 --> C1

    R1 --> Ingest
    R1 --> Index1
    R1 --> H1
    R1 --> T1
    R1 --> E1

    %% Click Events
    click Config1 "https://github.com/12parker/longcontextrag/blob/main/core/config.py"
    click Config2 "https://github.com/12parker/longcontextrag/blob/main/utils/setup_env.py"
    click Ingest "https://github.com/12parker/longcontextrag/blob/main/VectorDB/build_db.py"
    click VS1 "https://github.com/12parker/longcontextrag/tree/main/vector_store_eval/"
    click VS2 "https://github.com/12parker/longcontextrag/tree/main/vector_store_long/"
    click Index1 "https://github.com/12parker/longcontextrag/blob/main/core/index.py"
    click Index2 "https://github.com/12parker/longcontextrag/blob/main/core/long_context_config.py"
    click Index3 "https://github.com/12parker/longcontextrag/blob/main/core/prompts.py"
    click H1 "https://github.com/12parker/longcontextrag/blob/main/hybrid/hybrid_attention_rag.py"
    click H2 "https://github.com/12parker/longcontextrag/blob/main/hybrid/hybrid_rag_integration.py"
    click H3 "https://github.com/12parker/longcontextrag/blob/main/hybrid/working_hybrid_rag.py"
    click T1 "https://github.com/12parker/longcontextrag/blob/main/training/neural_retriever.py"
    click T2 "https://github.com/12parker/longcontextrag/blob/main/training/train_hybrid_rag.py"
    click E1 "https://github.com/12parker/longcontextrag/blob/main/evaluation/quick_rag_evaluation.py"
    click E2 "https://github.com/12parker/longcontextrag/blob/main/evaluation/rag_evaluation.py"
    click C1 "https://github.com/12parker/longcontextrag/blob/main/examples/compare_responses.py"
    click C2 "https://github.com/12parker/longcontextrag/blob/main/examples/examples.py"
    click C3 "https://github.com/12parker/longcontextrag/blob/main/examples/interactive_rag.py"
    click C4 "https://github.com/12parker/longcontextrag/blob/main/examples/long_context_testing.py"
    click C5 "https://github.com/12parker/longcontextrag/blob/main/examples/run_long_context_test.py"
    click C6 "https://github.com/12parker/longcontextrag/blob/main/main.py"
    click U1 "https://github.com/12parker/longcontextrag/blob/main/utils/test_setup.py"
    click U2 "https://github.com/12parker/longcontextrag/blob/main/utils/token_usage_analysis.py"
    click R1 "https://github.com/12parker/longcontextrag/blob/main/research/research_notebook.py"

    %% Styles
    classDef config fill:#D6EAF8,stroke:#3498DB;
    classDef ingest fill:#D5F5E3,stroke:#27AE60;
    classDef db fill:#FCF3CF,stroke:#F1C40F;
    classDef core fill:#AED6F1,stroke:#2980B9;
    classDef ext fill:#A9DFBF,stroke:#28B463;
    classDef external fill:#F1948A,stroke:#C0392B;
    classDef train fill:#F5CBA7,stroke:#D35400;
    classDef eval fill:#E8DAEF,stroke:#8E44AD;
    classDef client fill:#F9E79F,stroke:#B7950B;
    classDef util fill:#D2B4DE,stroke:#6C3483;
```
