How robot2050/attenhscore works
Project Type and Purpose
• This is a research-oriented, modular question-answering (QA) pipeline that implements an adaptive, collaborative paradigm between large language models (LLMs) and small language models (SLMs), with an emphasis on hallucination detection (AttenHScore) and retrieval-augmented generation (RAG).
• It provides data ingestion/chunking, vector indexing (Milvus), retrieval, LLM/SLM invocation, uncertainty-based reranking, and end-to-end evaluation on standard QA datasets.
High-Level File-Structure Analysis
• Top-level scripts and configs: run.sh, _settings.py, base.py, requirements.txt
• Data layer:
• data/ (raw JSON, JSONL, Parquet datasets, chunked outputs)
• dataeval/ (per-dataset evaluation loaders and metrics)
• Embeddings & Models:
• embeddings/ (base embedding interface)
• models/ (model-loading abstractions, adapters for NLI or OpenAI APIs)
• Core pipelines:
• retrieval.py & tollms_retrieval.py (RAG retrieval and simple LLM QA)
• pipeline/generate.py (LLM-SLM adaptive invocation logic)
• judge.py (central orchestration of collaborative QA)
• Supporting utilities:
• rerank.py, perplexity_chunking.py, metrics.py, func/ (plotting, evaluation helpers), utils/ (parallelization)
• Evaluation entrypoints: eval.py, func/evalFunc.py
Key README Insights
• Uses Milvus-Lite as vector store for passage retrieval.
• Plug-and-play for any transformer-based LMs.
• Two main usage flows:
a) SLM Retrieval-Based QA (retrieval.py)
b) Collaborative LLM-SLM QA (pipeline/generate.py + judge.py)
• Dedicated evaluation scripts measure hallucination detection and overall QA accuracy.
• Data Ingestion & Chunking
• Vector Database (Milvus-Lite)
• Retrieval Engine
• Embedding Service (embeddings/base.py)
• Model Abstraction Layer (models/_load_model.py + adapters)
• Adaptive Invocation Controller (AttenHScore logic in pipeline/generate.py)
• LLM Interface (tollms_retrieval.py)
• SLM Interface (retrieval.py)
• Reranker & Hallucination Detector (rerank.py, perplexity_chunking.py)
• Orchestrator (judge.py)
• Evaluation Suite (dataeval/, eval.py, func/evalFunc.py)
1) Raw QA Data → Chunking → Chunk JSON → Milvus Indexing
2) User Query → Retrieval Engine → Top-K Passages
3) Passages → Embedding Service → Vector DB lookup
4) Retrieved Context + Query → Adaptive Invocation Controller
– if uncertainty low → SLM Interface
– else → LLM Interface
5) Generated Answer → Reranker / Hallucination Detector
6) Chosen Answer → Orchestrator → Final Output
7) All outputs → Evaluation Suite for metrics plotting and analysis
c. Architectural Patterns & Principles:
• Modular pipeline with clear separation of concerns
• Plug-and-play model adapters for different LMs/APIs
• Asynchronous/parallel processing (utils/parallel.py) for speed
• Vector-search retrieval + reranking loop
• Decision logic (AttenHScore) driving conditional invocation
d. Key Technologies & Frameworks:
• Python with standard libs + Pandas/Parquet for data
• Milvus-Lite for vector storage
• Hugging Face Transformers (implied) & OpenAI API
• JSON/JSONL/Parquet I/O
• Jupyter notebook (judge.ipynb) for interactive analysis