How codex0870/nirf-college-ranking works
What this project is (type + purpose)
This repo is primarily a model packaging + inference usage project, not a traditional multi-service application.
• Core deliverable: a GGUF-converted, quantized Gemma 2B model fine-tuned for NIRF 2025 institutional ranking lookups.
• Primary runtime targets: llama.cpp, Ollama, and llama-cpp-python.
• Secondary/experimental workflows (from notebooks):
• Quantization / conversion experiments (gemma_quantization*.ipynb)
• RAG exploration (rag_1.ipynb, rag_2.ipynb, rag_3.ipynb)
• Web scraping (web_scrap.ipynb)
• Entry points / ops:
• use_gguf_model.sh (shell helper to run the GGUF model)
• main.py (likely a python driver for inference or a small demo CLI)
• requirements.txt (python deps, especially if using llama-cpp-python / RAG tooling)
Offline model build/publish pipeline (how GGUF + quantization are produced)
2) Online inference/usage architecture (how a user query flows through runtimes like llama.cpp/Ollama/python, and optionally RAG)
How to read the file tree as architecture signals
From the top-level files:
• README.md: defines the intended usage paths (llama.cpp, Ollama, python).
• gemma_quantization.ipynb, gemma_quantizationv2_0.ipynb: imply an offline pipeline stage (convert → quantize → validate).
• rag_*.ipynb: indicates an optional retrieval augmentation subsystem (scrape/index/retrieve → augment prompt → LLM).
• web_scrap.ipynb: suggests a data acquisition component feeding RAG or dataset building.
• main.py: likely a lightweight “app” wrapper (CLI or script) around llama-cpp-python.
• use_gguf_model.sh: OS-level wrapper to run inference locally.
Source model (HuggingFace)
• Base: google/gemma-2-2b-it
• Fine-tuned model: coderop12/gemma2b-nirf-lookup-2025 (QLoRA 4-bit)
Conversion tooling
• llama.cpp conversion scripts/tools (HuggingFace → GGUF F16)
Quantization step
• Quantize to Q4_K_M (recommended)
• gemma_quantization*.ipynb as the “orchestration / experimentation environment”
Published artifacts
• *.gguf files (F16 and Q4_K_M)
• Optionally “Modelfile” for Ollama import
Distribution
• HuggingFace repo / GitHub release / local artifact store (where users download from)
• HF Fine-tuned Model → (Convert) → GGUF F16 → (Quantize) → GGUF Q4_K_M → (Validate) → Publish
Online / Inference Usage Architecture (runtime-centric)
Core inference paths (three “clients”):
llama.cpp CLI path
• User → llama-cli → loads *.gguf → generates response
Ollama path
• User → ollama run → Ollama runtime → loads imported GGUF model → generates response
Python path (llama-cpp-python)
• User/App (main.py) → llama_cpp.Llama(...) → loads *.gguf → generates response
Shared resources:
• The GGUF model file on disk (artifact store)
• Compute resources: CPU/GPU (T4/RTX/etc.)
• User prompt → Runtime → Model → Token generation → Structured response
Optional RAG Subsystem (derived from rag_*.ipynb + web_scrap.ipynb)
Components:
Data acquisition
• web_scrap.ipynb → pulls NIRF pages / institutional info
Document store
• Raw HTML/text files (local disk / object store)
Embedding + indexing
• Embedding model (external dependency) → Vector DB / local FAISS index
Retriever
• Given user query → retrieve top-K docs/snippets
Prompt builder
• Combine user query + retrieved context + instruction template
LLM inference
• llama.cpp/Ollama/python runtime using GGUF model
Response
• Structured response with references
• Query → Retriever → Context → Prompt builder → LLM → Answer (+ citations)
• Actors: “User / Developer”
• External systems: HuggingFace, NIRF sources (if scraping), llama.cpp, Ollama
• System: “Gemma 2B NIRF Lookup GGUF package + scripts/notebooks”
• Artifacts: “GGUF model files”
• Containers: Notebooks (offline pipeline), Inference runtimes (llama.cpp / Ollama / Python app), Artifact storage, Optional RAG pipeline
• Documents → embeddings → retrieval → prompt assembly → generation
Relationships explicitly
• Artifact boundaries
• “Model artifact: gemma2b...q4_k_m.gguf” is a first-class node
• Runtime boundaries
• llama.cpp vs Ollama vs Python are separate execution environments
• Compute/memory expectations
• “~2GB VRAM Q4_K_M” / CPU option (a small annotation near runtimes)
• Source of truth
• “NIRF official sources” as external verification (even if not automated)
Make “optional” paths visually obvious
• RAG pipeline (since notebooks suggest exploration, not guaranteed production)
• Web scraping (optional data acquisition)
A “most accurate” component breakdown for this repo (don’t overthink; split it up)
Offline
• Fine-tuned HF model (QLoRA)
• Conversion step (HF → GGUF F16)
• Quantization step (F16 → Q4_K_M)
• Validation (NIRF queries)
• Published artifacts (GGUF files)
Online
• User
• Inference runtime (three alternatives: llama.cpp / Ollama / python)
• Local disk / artifact store containing GGUF
• Output formatter (structured response + references)
Optional RAG
• Web scraping notebook
• Raw corpus store
• Embedding model
• Vector index
• Retriever
• Prompt composer
• LLM inference (reuses online runtime)