How lomash-relia/qapipleline-using-llms works
Project type and purpose
• This is primarily a data-processing/ETL pipeline implemented as a Jupyter notebook (qa-llamacpp-pipeline.ipynb).
• Purpose: Ingest PDF books, run three sequential LLM “chains” (summarization, question generation, answer generation), apply a regex post-processor, and emit a structured dataset of summaries, questions, and answers.
File structure
• Top level:
• qa-llamacpp-pipeline.ipynb – the single notebook containing the entire pipeline
• kecs101.pdf – sample input file
• README.md, LICENSE, .gitignore – metadata
• No separate “src” or “lib” directories: all logic lives inline in the notebook.
• No external service configs (e.g., Dockerfile, CI scripts).
README for additional insights
• Confirms three “naive LLM chains” followed by regex.
• Notes LLMs are served via llamacpp (local Hugging Face models) on Kaggle or local.
Main components & interactions
a. Components
• Input Module: PDF Loader & text extractor
• Preprocessing Module: cleaning, chunking text
• Chain 1: Summarization LLM (via llamacpp)
• Chain 2: Question-Generation LLM
• Chain 3: Answer-Generation LLM
• Post-Processing: regex filter/formatter
• Output Module: dataset writer (e.g., CSV/JSON)
• Execution Environment: Kaggle VM or local Python environment with llamacpp
b. Relationships and data flow
• Preprocessing → Chain 1 → Chain 2 → Chain 3 → Post-Processing → Output
• Environment boundary around all LLM calls indicating llamaCPP as an external “model server”
c. Architectural patterns/principles
• Pipeline pattern (linear stages)
• Modularity: each chain is a separate stage that can be swapped or tuned independently
• External dependency: llamaCPP / Hugging Face models
d. Key technologies
• Python & Jupyter Notebook
• llamaCPP for model inference
• regex for post-processing
• PDF parsing library (e.g., PyPDF2 or pdfminer)
Detailed capture of essential elements
• Be explicit about how raw PDF text becomes chained LLM prompts and responses.
• That each LLM chain takes the previous stage’s output as input.
• Regex step as the final transformation before writing output.
• Performance considerations or estimated runtimes per stage if relevant.