How 12parker/longcontextrag works
Project type and purpose
• This is a research framework for “Retrieval Augmented Generation” (RAG) optimized for very long contexts (up to 128k+ tokens). It combines an external vector store (ChromaDB) with large language models (via OpenAI API) and provides hybrid attention modules, evaluation pipelines, training scripts, and example clients.
File structure
• Top-level modules:
• core/ – core RAG system (config, indexing, prompt templates, long-context handling)
• hybrid/ – hybrid attention RAG extension (neural retriever, attention mechanisms, integration)
• VectorDB/ – vector database builder for BookCorpus (build_db.py)
• evaluation/ – evaluation framework (quick and full evaluations)
• examples/ – demo and interactive scripts
• training/ – scripts to train neural retrievers and hybrid RAG
• testing/ – test harnesses for BookCorpus integration and hybrid tests
• utils/ – environment setup, test setup, token usage analysis
• research/ – notebooks and experimentation tools
• Deployment/config:
• activate.sh, env.template, requirements.txt
• main.py – entry point for ad hoc runs
• Data directories (persisted outputs):
• vector_store_eval/, vector_store_long/ – serialized ChromaDB stores
README for additional insights
• External services: OpenAI API, ChromaDB
• Core features: chunking, TOP_K retrieval, configurable context window, multiple prompt templates, hybrid sliding/sparse attention strategies
• Usage patterns: build vector store → index documents → generate response with or without retrieval → evaluate performance