# xndien2004/vidrill architecture

> Architecture diagram of the GitHub repository xndien2004/vidrill, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/xndien2004/vidrill
- Repository: https://github.com/xndien2004/vidrill

## Overview

##### Project Type & Purpose
- Project Type: A multi-stage document retrieval system with optional model training and evaluation subsystems.
- Purpose: Indexed retrieval over large Vietnamese legal corpora using hybrid methods (dense embeddings + BM25) plus optional LLM-based rewriting, selection, and reranking for future pipelines.

##### File Structure
- Top-Level Modules:
– prepare_data/ : corpus preprocessing, chunking, hard-negative mining, DB building
– pipeline/      : end-to-end retrieval (encoding, search, hybrid scoring)
– training/      : subfolders for retrieval, rerank, LLM fine-tuning, DeepRetrieval, ViSearch-R1
– eval/          : evaluation scripts and metrics
– config/        : JSON configs for DeepSpeed, PPO/GRPO training
- Root Scripts:
– encoder.py, search.py, conbine_ranking_score.py handle embedding, querying & result combination
- Scripts Folders: orchestration bash scripts under pipeline/scripts, prepare_data/scripts, training/*/scripts

##### Extract Architectural Insights from README
- Multi-stage Hybrid Pipeline:
– Sparse retrieval (BM25) + Dense retrieval (E5, GTE, BGE-M3 embeddings)
– Neural reranking (BGE reranker)
– Future LLM modules for query rewriting & selection
- Scalable Vector DB: uses Qdrant to store and query embeddings
- Modular Design: clear separation between retrieval, rerank, data prep, training, and eval

##### Main Components
A. Data Ingestion & Preprocessing (“prepare_data”)
– Inputs: raw legal documents
– Processes: chunking, building BM25 index, hard-negative selection, corpus encoder writes to vector DB
B. Storage & Indexing
– Qdrant Vector DB (dense embeddings)
– BM25 inverted index
C. Retrieval Pipeline (“pipeline”)
– Corpus Encoder Service (encoder.py)
– Query Processing: optional HyDE prompt generation (hyde_llm.py)
– Dense Retrieval Engines: E5 + Sentence-BERT modules (main_e5_sentence.py)
– Hybrid Scoring Module (conbine_ranking_score.py)
– BM25 Retriever (via search.py)
– Final Hit Aggregation & Filtering
D. Reranking
– BGE Reranker model invoked post-retrieval
– Optional LLM re-ranking paths
E. Model Training Subsystems (“training”)
– retrieval/: training scripts for E5, GTE, BGE-M3
– rerank/: training BGE reranker
– llm/: scripts & code for SFT & GRPO fine-tuning, plus prompt/reward modules
– DeepRetrieval/ & ViSearch-R1/: RL-based query rewriting & self-searching training servers
F. Evaluation (“eval”)
– Scripts that invoke retrieval pipeline, compute metrics

##### Relationships & Data Flows
- Pipeline queries DB & index → returns candidate docs → reranker → final results
- Training subsystems read from prepared corpora & ground-truth labels → update model artifacts (checkpoint outputs)
- Eval jobs call pipeline components & produce metric outputs

##### Architectural Patterns & Principles
- Modular, stage-based pipeline (Separation of Concerns)
- Pluggable retrieval heads (allow E5, GTE, BM25, LLM modules to be added/removed)
- Config-driven behavior (config/*.json for DeepSpeed & PPO settings)
- Scalability via external vector DB (Qdrant) and shell-script orchestration

##### Technologies & Frameworks
- Python services & scripts
- Qdrant (vector storage)
- Hugging Face transformers for embeddings & LLM
- BM25 (e.g., via Whoosh or Elastic/Pyserini)
- DeepSpeed ZeRO for large-model training
- RL libraries for PPO/GRPO in LLM training

##### Final Checklist
[ ] All main components represented
[ ] Data flows and dependencies are directional and unambiguous
[ ] Architectural patterns (modular stages, config-driven) are annotated
[ ] External services (Qdrant, Hugging Face APIs) are shown
[ ] Optional/future LLM blocks are clearly marked

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Prep Plane
    subgraph "Data Prep Plane"
        direction TB
        RawDocs["Raw Legal Documents"]:::service
        Chunking["chunking.py"]:::service
        BuildDB["build_db_corpus.py"]:::service
        HardNeg["hard_neg.py"]:::service
        ChoiceNeg["choice_hard_neg.py"]:::service
        CheckTrainPrep["check_train.py"]:::service
        PrepScripts["prepare_data/scripts/"]:::service
        VectorDB["Qdrant Vector DB"]:::datastore
        BM25Index["BM25 Inverted Index"]:::datastore
    end

    RawDocs -->|chunk| Chunking
    Chunking -->|build corpus| BuildDB
    BuildDB --> VectorDB
    BuildDB --> BM25Index
    Chunking --> HardNeg
    HardNeg --> ChoiceNeg
    ChoiceNeg --> CheckTrainPrep
    CheckTrainPrep --> PrepScripts

    %% Retrieval & Reranking Plane
    subgraph "Retrieval & Reranking Plane"
        direction TB
        UserQuery["User Query"]:::service
        HyDE["hyde_llm.py\n(future)"]:::future
        CorpusEncService["corpus_encoder.py"]:::service
        EncoderRoot["encoder.py"]:::service
        BM25Retriever["search.py"]:::service
        DenseE5["main_e5_sentence.py"]:::service
        DenseE5BM25["main_e5_sentence_bm25.py"]:::service
        HybridScore["conbine_ranking_score.py"]:::service
        LLMinfer["llm_infer.py\n(future)"]:::future
        InferScripts["pipeline/scripts/"]:::service
        Reranker["BGE Reranker"]:::service
        FinalResults["Final Results"]:::service
    end

    UserQuery --> HyDE
    UserQuery --> CorpusEncService
    UserQuery --> BM25Retriever
    CorpusEncService --> DenseE5
    CorpusEncService --> DenseE5BM25
    DenseE5 --> HybridScore
    DenseE5BM25 --> HybridScore
    BM25Retriever --> HybridScore
    HybridScore --> Reranker
    Reranker --> FinalResults
    FinalResults --> LLMinfer
    LLMinfer --> InferScripts

    %% Training & Eval Plane
    subgraph "Training & Eval Plane"
        direction TB
        PrepData["Prepared Corpora & Labels"]:::service
        TrainRetrieval["training/retrieval/"]:::training
        TrainRerank["training/rerank/"]:::training
        TrainLLM["training/llm/"]:::training
        DeepRetrieval["training/DeepRetrieval/"]:::training
        ViSearchR1["training/ViSearch-R1/"]:::training
        ConfigFiles["config/"]:::service
        EvalScripts["eval/"]:::eval
        EvalJob["eval.py & scripts"]:::eval
    end

    VectorDB --> PrepData
    BM25Index --> PrepData
    PrepData --> TrainRetrieval
    PrepData --> TrainRerank
    PrepData --> TrainLLM
    PrepData --> DeepRetrieval
    PrepData --> ViSearchR1
    ConfigFiles --> TrainLLM
    EvalJob --> FinalResults

    %% Click Events
    click Chunking "https://github.com/xndien2004/vidrill/blob/main/prepare_data/chunking.py"
    click BuildDB "https://github.com/xndien2004/vidrill/blob/main/prepare_data/build_db_corpus.py"
    click HardNeg "https://github.com/xndien2004/vidrill/blob/main/prepare_data/hard_neg.py"
    click ChoiceNeg "https://github.com/xndien2004/vidrill/blob/main/prepare_data/choice_hard_neg.py"
    click CheckTrainPrep "https://github.com/xndien2004/vidrill/blob/main/prepare_data/check_train.py"
    click PrepScripts "https://github.com/xndien2004/vidrill/tree/main/prepare_data/scripts/"
    click BM25Retriever "https://github.com/xndien2004/vidrill/blob/main/search.py"
    click CorpusEncService "https://github.com/xndien2004/vidrill/blob/main/pipeline/corpus_encoder.py"
    click EncoderRoot "https://github.com/xndien2004/vidrill/blob/main/encoder.py"
    click DenseE5 "https://github.com/xndien2004/vidrill/blob/main/pipeline/main_e5_sentence.py"
    click DenseE5BM25 "https://github.com/xndien2004/vidrill/blob/main/pipeline/main_e5_sentence_bm25.py"
    click HyDE "https://github.com/xndien2004/vidrill/blob/main/pipeline/hyde_llm.py"
    click LLMinfer "https://github.com/xndien2004/vidrill/blob/main/pipeline/llm_infer.py"
    click InferScripts "https://github.com/xndien2004/vidrill/tree/main/pipeline/scripts/"
    click HybridScore "https://github.com/xndien2004/vidrill/blob/main/conbine_ranking_score.py"
    click TrainRetrieval "https://github.com/xndien2004/vidrill/tree/main/training/retrieval/"
    click TrainRerank "https://github.com/xndien2004/vidrill/tree/main/training/rerank/"
    click TrainLLM "https://github.com/xndien2004/vidrill/tree/main/training/llm/"
    click DeepRetrieval "https://github.com/xndien2004/vidrill/tree/main/training/DeepRetrieval/"
    click ViSearchR1 "https://github.com/xndien2004/vidrill/tree/main/training/ViSearch-R1/"
    click EvalScripts "https://github.com/xndien2004/vidrill/tree/main/eval/"
    click EvalJob "https://github.com/xndien2004/vidrill/blob/main/eval/eval.py"
    click ConfigFiles "https://github.com/xndien2004/vidrill/tree/main/config/"

    %% Styles
    classDef datastore fill:#ADD8E6,stroke:#000
    classDef service fill:#90EE90,stroke:#000
    classDef training fill:#FFA500,stroke:#000
    classDef eval fill:#D3D3D3,stroke:#000
    classDef future fill:#F0E68C,stroke:#000
```
