How coderonfleek/rag-from-scratch works
This repository is a command-line RAG demo. Its build workflow reads supported local files, stores their contents in SQLite, chunks them, embeds them, and persists vectors in Chroma. Its query workflow loads that vector store, retrieves top-k documents for a supplied query, and prints them; it does not generate an LLM answer. The README also documents HuggingFace, OpenAI, and Google embedding options. The workflow is supported by the sampled source; optional provider connections are shown as integrations.
Main entry points: main.py CLI orchestrator · ingest.py File ingestion · pipeline.py Persistent Chroma · db.py Raw document store
Components 13
CLI workflows
- CLI orchestrator
rag_pipeline/main.py - Printed documents
rag_pipeline/main.py
Document ingestion
- File ingestion
rag_pipeline/ingest.py
Chunking and retrieval
- Document chunking
rag_pipeline/pipeline.py - Top-k retrieval
rag_pipeline/pipeline.py
Persistence and embeddings
- Raw document store
rag_pipeline/db.py - Embedding selection
rag_pipeline/pipeline.py - Vector store build
rag_pipeline/pipeline.py - Persistent Chroma
rag_pipeline/pipeline.py
Other components
- CLI user
- Local documents
- Search query
- Embedding providers
Connections 15
- CLI user → CLI orchestrator: runs
- Local documents → File ingestion: reads
- Search query → CLI orchestrator: provides query
- CLI orchestrator → File ingestion: calls build
- File ingestion → Raw document store: initializes and writes
- CLI orchestrator → Raw document store: fetches documents
- CLI orchestrator → Document chunking: chunks documents
- CLI orchestrator → Embedding selection: selects embeddings
- Embedding selection → Embedding providers: constructs provider
- CLI orchestrator → Vector store build: builds vectors
- Vector store build → Persistent Chroma: persists vectors
- CLI orchestrator → Persistent Chroma: loads store
- CLI orchestrator → Top-k retrieval: gets retriever and invokes
- Top-k retrieval → Persistent Chroma: searches vectors
- CLI orchestrator → Printed documents: prints retrieved documents