# robot2050/attenhscore architecture

> Architecture diagram of the GitHub repository robot2050/attenhscore, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/robot2050/attenhscore
- Repository: https://github.com/robot2050/attenhscore

## Overview

##### Project Type and Purpose
- This is a research-oriented, modular question-answering (QA) pipeline that implements an adaptive, collaborative paradigm between large language models (LLMs) and small language models (SLMs), with an emphasis on hallucination detection (AttenHScore) and retrieval-augmented generation (RAG).
- It provides data ingestion/chunking, vector indexing (Milvus), retrieval, LLM/SLM invocation, uncertainty-based reranking, and end-to-end evaluation on standard QA datasets.

##### High-Level File-Structure Analysis
- Top-level scripts and configs: run.sh, _settings.py, base.py, requirements.txt
- Data layer:
- data/ (raw JSON, JSONL, Parquet datasets, chunked outputs)
- dataeval/ (per-dataset evaluation loaders and metrics)
- Embeddings & Models:
- embeddings/ (base embedding interface)
- models/ (model-loading abstractions, adapters for NLI or OpenAI APIs)
- Core pipelines:
- retrieval.py & tollms_retrieval.py (RAG retrieval and simple LLM QA)
- pipeline/generate.py (LLM-SLM adaptive invocation logic)
- judge.py (central orchestration of collaborative QA)
- Supporting utilities:
- rerank.py, perplexity_chunking.py, metrics.py, func/ (plotting, evaluation helpers), utils/ (parallelization)
- Evaluation entrypoints: eval.py, func/evalFunc.py

##### Key README Insights
- Uses Milvus-Lite as vector store for passage retrieval.
- Plug-and-play for any transformer-based LMs.
- Two main usage flows:
a) SLM Retrieval-Based QA (retrieval.py)
b) Collaborative LLM-SLM QA (pipeline/generate.py + judge.py)
- Dedicated evaluation scripts measure hallucination detection and overall QA accuracy.

- Data Ingestion & Chunking
- Vector Database (Milvus-Lite)
- Retrieval Engine
- Embedding Service (embeddings/base.py)
- Model Abstraction Layer (models/_load_model.py + adapters)
- Adaptive Invocation Controller (AttenHScore logic in pipeline/generate.py)
- LLM Interface (tollms_retrieval.py)
- SLM Interface (retrieval.py)
- Reranker & Hallucination Detector (rerank.py, perplexity_chunking.py)
- Orchestrator (judge.py)
- Evaluation Suite (dataeval/, eval.py, func/evalFunc.py)

1) Raw QA Data → Chunking → Chunk JSON → Milvus Indexing
2) User Query → Retrieval Engine → Top-K Passages
3) Passages → Embedding Service → Vector DB lookup
4) Retrieved Context + Query → Adaptive Invocation Controller
– if uncertainty low → SLM Interface
– else → LLM Interface
5) Generated Answer → Reranker / Hallucination Detector
6) Chosen Answer → Orchestrator → Final Output
7) All outputs → Evaluation Suite for metrics plotting and analysis

c. Architectural Patterns & Principles:
- Modular pipeline with clear separation of concerns
- Plug-and-play model adapters for different LMs/APIs
- Asynchronous/parallel processing (utils/parallel.py) for speed
- Vector-search retrieval + reranking loop
- Decision logic (AttenHScore) driving conditional invocation

d. Key Technologies & Frameworks:
- Python with standard libs + Pandas/Parquet for data
- Milvus-Lite for vector storage
- Hugging Face Transformers (implied) & OpenAI API
- JSON/JSONL/Parquet I/O
- Jupyter notebook (judge.ipynb) for interactive analysis

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration & Startup
    subgraph "Config & Startup"
        Settings["_settings.py"]:::compute
        Base["base.py"]:::compute
        Run["run.sh"]:::compute
        Req["requirements.txt"]:::compute
    end

    %% Data Layer
    subgraph "Data Layer"
        DataIngest["Data Ingestion & Chunking"]:::dataStorage
        Milvus["Milvus-Lite DB"]:::external
    end

    %% Model Layer
    subgraph "Model Layer"
        EmbService["Embedding Service"]:::compute
        ModelLayer["Model Abstraction Layer"]:::compute
        NLI["NLI Adapter"]:::compute
        OpenAI["OpenAI Adapter"]:::compute
    end

    %% Pipeline
    subgraph "Pipeline"
        User["User Query"]:::external
        RetrievalEngine["Retrieval Engine"]:::compute
        Decision{"Use LLM or SLM?"}:::control
        SLMInterface["SLM Interface"]:::compute
        LLMInterface["LLM Interface"]:::compute
        GeneratedAnswer["Generated Answer"]:::compute
        Reranker["Reranker"]:::compute
        Perplexity["Hallucination Detector"]:::compute
        ChosenAnswer["Chosen Answer"]:::compute
        Orchestrator["Orchestrator"]:::compute
        FinalOutput["Final Output"]:::compute
    end

    %% Evaluation Suite
    subgraph "Evaluation Suite"
        DataEval["Data-specific Evaluators"]:::dataStorage
        EvalEntrypoint["eval.py"]:::compute
        EvalFuncs["Evaluation Helpers"]:::compute
    end

    %% Infrastructure
    subgraph "Infrastructure"
        Parallel["Parallelization Utils"]:::external
    end

    %% Connections
    Settings --> DataIngest
    Run --> DataIngest
    Base --> DataIngest
    Req --> DataIngest

    DataIngest -->|"JSONL chunks"| Milvus

    User --> RetrievalEngine
    RetrievalEngine --> EmbService
    EmbService -->|"Embedding vectors"| Milvus
    Milvus -->|"Top-K IDs"| RetrievalEngine

    RetrievalEngine -->|"Top-K Passages"| Decision
    Decision -->|"uncertainty low"| SLMInterface
    Decision -->|"uncertainty high"| LLMInterface

    SLMInterface --> GeneratedAnswer
    LLMInterface --> GeneratedAnswer

    GeneratedAnswer --> Reranker
    GeneratedAnswer --> Perplexity
    Reranker --> ChosenAnswer
    Perplexity --> ChosenAnswer

    ChosenAnswer --> Orchestrator
    Orchestrator --> FinalOutput
    FinalOutput --> DataEval
    FinalOutput --> EvalEntrypoint
    FinalOutput --> EvalFuncs

    Parallel -.-> RetrievalEngine
    Parallel -.-> Decision

    %% Click Events
    click Settings "https://github.com/robot2050/attenhscore/blob/main/_settings.py"
    click Base "https://github.com/robot2050/attenhscore/blob/main/base.py"
    click Run "https://github.com/robot2050/attenhscore/blob/main/run.sh"
    click Req "https://github.com/robot2050/attenhscore/blob/main/requirements.txt"
    click DataIngest "https://github.com/robot2050/attenhscore/tree/main/data/chunking/"
    click Milvus "https://github.com/robot2050/attenhscore/blob/main/retrieval.py"
    click RetrievalEngine "https://github.com/robot2050/attenhscore/blob/main/retrieval.py"
    click EmbService "https://github.com/robot2050/attenhscore/blob/main/embeddings/base.py"
    click ModelLayer "https://github.com/robot2050/attenhscore/blob/main/models/_load_model.py"
    click NLI "https://github.com/robot2050/attenhscore/blob/main/models/nli.py"
    click OpenAI "https://github.com/robot2050/attenhscore/blob/main/models/openai_models.py"
    click Decision "https://github.com/robot2050/attenhscore/blob/main/pipeline/generate.py"
    click LLMInterface "https://github.com/robot2050/attenhscore/blob/main/tollms_retrieval.py"
    click SLMInterface "https://github.com/robot2050/attenhscore/blob/main/retrieval.py"
    click Reranker "https://github.com/robot2050/attenhscore/blob/main/rerank.py"
    click Perplexity "https://github.com/robot2050/attenhscore/blob/main/perplexity_chunking.py"
    click Orchestrator "https://github.com/robot2050/attenhscore/blob/main/judge.py"
    click DataEval "https://github.com/robot2050/attenhscore/tree/main/dataeval/"
    click EvalEntrypoint "https://github.com/robot2050/attenhscore/blob/main/eval.py"
    click EvalFuncs "https://github.com/robot2050/attenhscore/blob/main/func/evalFunc.py"
    click Parallel "https://github.com/robot2050/attenhscore/blob/main/utils/parallel.py"

    %% Styles
    classDef dataStorage fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef compute fill:#90EE90,stroke:#000,stroke-width:1px
    classDef control fill:#FFA500,stroke:#000,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#000,stroke-width:1px
```
