# lomash-relia/qapipleline-using-llms architecture

> Architecture diagram of the GitHub repository lomash-relia/qapipleline-using-llms, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lomash-relia/qapipleline-using-llms
- Repository: https://github.com/lomash-relia/qapipleline-using-llms

## Overview

##### Project type and purpose
- This is primarily a data-processing/ETL pipeline implemented as a Jupyter notebook (qa-llamacpp-pipeline.ipynb).
- Purpose: Ingest PDF books, run three sequential LLM “chains” (summarization, question generation, answer generation), apply a regex post-processor, and emit a structured dataset of summaries, questions, and answers.

##### File structure
- Top level:
- qa-llamacpp-pipeline.ipynb – the single notebook containing the entire pipeline
- kecs101.pdf – sample input file
- README.md, LICENSE, .gitignore – metadata
- No separate “src” or “lib” directories: all logic lives inline in the notebook.
- No external service configs (e.g., Dockerfile, CI scripts).

##### README for additional insights
- Confirms three “naive LLM chains” followed by regex.
- Notes LLMs are served via llamacpp (local Hugging Face models) on Kaggle or local.

##### Main components & interactions
a. Components
- Input Module: PDF Loader & text extractor
- Preprocessing Module: cleaning, chunking text
- Chain 1: Summarization LLM (via llamacpp)
- Chain 2: Question-Generation LLM
- Chain 3: Answer-Generation LLM
- Post-Processing: regex filter/formatter
- Output Module: dataset writer (e.g., CSV/JSON)
- Execution Environment: Kaggle VM or local Python environment with llamacpp
b. Relationships and data flow
- Preprocessing → Chain 1 → Chain 2 → Chain 3 → Post-Processing → Output
- Environment boundary around all LLM calls indicating llamaCPP as an external “model server”
c. Architectural patterns/principles
- Pipeline pattern (linear stages)
- Modularity: each chain is a separate stage that can be swapped or tuned independently
- External dependency: llamaCPP / Hugging Face models
d. Key technologies
- Python & Jupyter Notebook
- llamaCPP for model inference
- regex for post-processing
- PDF parsing library (e.g., PyPDF2 or pdfminer)

##### Detailed capture of essential elements
- Be explicit about how raw PDF text becomes chained LLM prompts and responses.
- That each LLM chain takes the previous stage’s output as input.
- Regex step as the final transformation before writing output.
- Performance considerations or estimated runtimes per stage if relevant.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Execution Environment (Kaggle/Local)"
        direction TB
        PDFFile["Sample Input (PDF Book)"]:::dataModule
        Loader["PDF Loader & Text Extractor"]:::dataModule
        Preprocessor["Preprocessing (Cleaning & Chunking)"]:::dataModule
        Summarize["Chain 1: Summarization LLM"]:::LLM
        QGen["Chain 2: Question-Generation LLM"]:::LLM
        AGen["Chain 3: Answer-Generation LLM"]:::LLM
        PostProc["Post-Processing (Regex Filter/Formatter)"]:::dataModule
        Output["Output Module (CSV/JSON Writer)"]:::dataModule
        DiagramAssets["Diagram Assets"]:::dataModule
        EnvNotes["Execution Environment Notes"]:::dataModule
    end

    llamaCPP["llamaCPP Runtime"]:::external

    PDFFile --> Loader
    Loader --> Preprocessor
    Preprocessor --> Summarize
    Summarize --> QGen
    QGen --> AGen
    AGen --> PostProc
    PostProc --> Output

    Summarize -->|"calls"| llamaCPP
    QGen -->|"calls"| llamaCPP
    AGen -->|"calls"| llamaCPP

    click Loader "https://github.com/lomash-relia/qapipleline-using-llms/blob/main/qa-llamacpp-pipeline.ipynb"
    click Preprocessor "https://github.com/lomash-relia/qapipleline-using-llms/blob/main/qa-llamacpp-pipeline.ipynb"
    click Summarize "https://github.com/lomash-relia/qapipleline-using-llms/blob/main/qa-llamacpp-pipeline.ipynb"
    click QGen "https://github.com/lomash-relia/qapipleline-using-llms/blob/main/qa-llamacpp-pipeline.ipynb"
    click AGen "https://github.com/lomash-relia/qapipleline-using-llms/blob/main/qa-llamacpp-pipeline.ipynb"
    click PostProc "https://github.com/lomash-relia/qapipleline-using-llms/blob/main/qa-llamacpp-pipeline.ipynb"
    click Output "https://github.com/lomash-relia/qapipleline-using-llms/blob/main/qa-llamacpp-pipeline.ipynb"
    click PDFFile "https://github.com/lomash-relia/qapipleline-using-llms/blob/main/kecs101.pdf"
    click DiagramAssets "https://github.com/lomash-relia/qapipleline-using-llms/tree/main/images/"
    click EnvNotes "https://github.com/lomash-relia/qapipleline-using-llms/blob/main/README.md"

    classDef dataModule fill:#add8e6,stroke:#333,stroke-width:1px
    classDef LLM fill:#90ee90,stroke:#333,stroke-width:1px
    classDef external fill:#ffcc99,stroke:#333,stroke-width:1px
```
