How abhi-sys511/llama3-rag-knowledge-assistant works
1. This repository is a local Retrieval-Augmented Generation (RAG) research assistant, centered on a CLI workflow rather than a web app. The primary path is: user input → document retrieval → prompt construction → local LLM generation → streamed response → structured logging.
2. main.py is described in the README as the main controller. It likely owns the interactive loop, command parsing, timing, streaming, and the handoff between retrieval and generation. It is the orchestration boundary, not the knowledge layer itself.
3. rag_engine.py is the core “memory” subsystem. It handles PDF ingestion, text extraction, chunking, embedding creation, and persistence into a Chroma vector store. It also provides retrieval: converting a question into a semantic search against stored chunks and returning the most relevant context.
4. rag.py is the prompt-augmentation layer. Its job is to merge retrieved document context with the user’s question into the final prompt sent to the LLM. This is the boundary that turns raw retrieval into grounded generation.
5. The local model runtime is Ollama with Llama3, as stated in the README. Python does not call a remote API; instead it talks to the local Ollama service, which loads the model and streams tokens back. That makes the system offline, private, and dependent on local CPU/GPU capacity rather than cloud billing.
6. Persistent knowledge storage is a Chroma vector database under knowledge_db/ per the README. That directory is the long-term semantic memory: embedded chunks and metadata are stored there so PDFs do not need to be reprocessed on every run.
7. The document ingestion flow is multi-step and important for any architecture diagram: PDF file → text extraction → chunk splitting → embeddings → vector DB persistence. The sample PDF ABAP_New.pdf in the repo is likely used as a test document for this pipeline.
8. The query-time flow is the mirror image: user question → embedding/search → top matching chunks → prompt assembly in rag.py → Ollama/Llama3 generation → streamed answer in the terminal. This is the central RAG loop that reduces hallucination by grounding the answer in loaded documents.
9. Logging and observability are first-class concerns here. The README describes structured JSON output with timestamp, query, response, and response time; the file tree shows research_output.txt, suggesting the logging artifact may have changed over time. Either way, the architecture includes response telemetry as a separate concern from generation.
10. evaluate.py appears to be an analysis or benchmarking utility, likely used to assess responses, performance, or retrieval quality. tools.py is probably shared helper code for file, text, or logging utilities. These are support modules around the main pipeline, not core runtime dependencies.
11. app.py is present alongside main.py, so there are likely multiple entrypoints or an earlier wrapper layer. For diagram purposes, treat main.py as the interactive controller and app.py as either an alternate launcher or packaging entrypoint unless code inspection shows otherwise.
12. The architectural boundaries are clear: controller layer (main.py/app.py), retrieval/memory layer (rag_engine.py + Chroma storage), prompt layer (rag.py), model runtime (Ollama/Llama3), and observability/output (research_output.*, timing, evaluation). That separation is what makes the project modular and diagram-friendly.
Main entry points: main.py Main loop · rag_engine.py RAG engine · tools.py Tools · rag.py Prompt builder
Components 14
Entry points
Command-line and launch surfaces
RAG pipeline
Ingestion, retrieval, and prompt assembly
- RAG engine · PDF ingestion, chunking, embeddings, persistence, and retrieval
rag_engine.py - Prompt builder · Merges retrieved context with the user question
rag.py - Ingest flow · Text extraction, splitting, and embedding preparation
- Search flow · Semantic lookup of relevant chunks for a query
Model runtime
Local generation backend
- Llama3 · Model loaded through the local Ollama service
- Ollama · Local token-streaming runtime used for generation
Knowledge store
Persistent semantic memory and outputs
- Chroma DB · Persistent chunk and metadata storage for long-term memory
- Research log · Structured response timing and result logging
research_output.txt - Sample PDF · Example knowledge document used to seed retrieval
ABAP_New.pdf
Connections 19
- App → Main loop: launches · Alternate entrypoint delegates into the CLI controller
- Main loop → RAG engine: ingest/retrieve · Controller calls the memory subsystem for indexing and search
- Main loop → Prompt builder: build prompt · Controller passes retrieved context into prompt augmentation
- Main loop → Ollama: generate · Controller sends the final prompt to the local model runtime
- Ollama → Llama3: hosts · Ollama loads and serves the Llama3 model locally
- Main loop → Streamed answer: streams · Generated tokens are emitted back to the terminal
- Main loop → Research log: logs · Query, response, and timing are recorded as telemetry
- Sample PDF → Ingest flow: source · PDF content is the document input for ingestion
- Ingest flow → RAG engine: indexes · Ingestion pipeline is implemented by the RAG engine
- RAG engine → Chroma DB: persists · Embedded chunks are stored in the vector database
- Main loop → Search flow: asks · User questions trigger retrieval against the knowledge base
- Search flow → RAG engine: searches · Semantic search returns the most relevant chunks
- RAG engine → Chroma DB: queries · Retrieval reads from persisted embeddings and metadata
- Search flow → Prompt builder: supplies context · Top matches are fed into prompt construction
- Prompt builder → Ollama: format prompt · Augmented prompt is handed to the local model host
- Evaluate → Research log: analyzes · Evaluation likely consumes stored run outputs and timings
- Tools → Main loop: supports · Shared helper functions support the CLI workflow
- Tools → RAG engine: supports · Utilities assist file and text handling in the memory pipeline
- Tools → Evaluate: supports · Utilities may be reused by analysis workflows