How maxkrnl/prompt-tester works
Project Type & Purpose
• This is a Python CLI demo illustrating a two-phase RAG pipeline:
a. Index Building (one-time precompute of FAISS index)
b. Interactive Generation (load index + quantized Llama model to produce tweets or replies)
• No web frontend or database server—pure offline, file-based storage and local model inference.
File Structure
Top-level scripts and directories:
• company_data/ – raw .txt knowledge files
• build_rag_index.py – ingestion, chunking, embedding, FAISS index creation
• rag_utils_demo.py – loading FAISS index + retrieval logic
• llm_utils_demo.py – instantiating 8-bit quantized Llama, prompt building, output cleaning
• demo.py – CLI interface orchestrating RAG retrieval + model inference
• Artifacts: faiss_index.bin, doc_texts.pkl (index + chunk store)
README Insights
• Precompute: run build_rag_index.py once after updating company_data/.
• Dependencies: Python 3.8+, FAISS, Huggingface Transformers, quantization libs.
• Runtime Flow:
1. CLI menu → choice: new tweet or reply
2. Query passes into rag_utils_demo → retrieve top-k chunks
3. Retrieved context + user prompt passed to llm_utils_demo → text generation
Defining the Main Components
a. Ingestion & Indexing Module
• Input: raw .txt files in company_data/
• Processes: chunking → embedding (e.g., via Huggingface embeddings) → FAISS index creation
• Output: faiss_index.bin, doc_texts.pkl
b. Retrieval Module
• Input: user query, FAISS index on disk
• Process: similarity search → retrieve top-k document chunks
• Output: context snippets
c. Model Inference Module
• Input: context snippets + formatted prompt
• Process: load quantized Llama (8-bit) via transformers → generate text
• Output: raw Llama output → cleaned reply/tweet text
d. CLI Orchestrator
• Presents menu, reads user input, calls Retrieval → Model Inference → displays output
e. External Dependencies
• Huggingface Hub (model download & login)
• Local filesystem for index and artifacts
Relationships & Data Flows
• From company_data/ → Indexing Module → faiss_index.bin + doc_texts.pkl
• CLI Orchestrator → Retrieval Module → Model Inference Module → CLI Output
• External: Model weights fetched from Huggingface Hub before first inference
• Data store boundaries: separate build-time vs. runtime artifacts
Architectural Patterns & Principles
• Modular separation: indexing, retrieval, inference, and orchestration each in its own script/module
• File-based persistence: FAISS index + pickled chunks ensure fast startup
• Single responsibility: each module does one thing well