How xndien2004/vidrill works
Project Type & Purpose
• Project Type: A multi-stage document retrieval system with optional model training and evaluation subsystems.
• Purpose: Indexed retrieval over large Vietnamese legal corpora using hybrid methods (dense embeddings + BM25) plus optional LLM-based rewriting, selection, and reranking for future pipelines.
File Structure
• Top-Level Modules:
– prepare_data/ : corpus preprocessing, chunking, hard-negative mining, DB building
– pipeline/ : end-to-end retrieval (encoding, search, hybrid scoring)
– training/ : subfolders for retrieval, rerank, LLM fine-tuning, DeepRetrieval, ViSearch-R1
– eval/ : evaluation scripts and metrics
– config/ : JSON configs for DeepSpeed, PPO/GRPO training
• Root Scripts:
– encoder.py, search.py, conbine_ranking_score.py handle embedding, querying & result combination
• Scripts Folders: orchestration bash scripts under pipeline/scripts, prepare_data/scripts, training/*/scripts
Extract Architectural Insights from README
• Multi-stage Hybrid Pipeline:
– Sparse retrieval (BM25) + Dense retrieval (E5, GTE, BGE-M3 embeddings)
– Neural reranking (BGE reranker)
– Future LLM modules for query rewriting & selection
• Scalable Vector DB: uses Qdrant to store and query embeddings
• Modular Design: clear separation between retrieval, rerank, data prep, training, and eval
Main Components
A. Data Ingestion & Preprocessing (“prepare_data”)
– Inputs: raw legal documents
– Processes: chunking, building BM25 index, hard-negative selection, corpus encoder writes to vector DB
B. Storage & Indexing
– Qdrant Vector DB (dense embeddings)
– BM25 inverted index
C. Retrieval Pipeline (“pipeline”)
– Corpus Encoder Service (encoder.py)
– Query Processing: optional HyDE prompt generation (hyde_llm.py)
– Dense Retrieval Engines: E5 + Sentence-BERT modules (main_e5_sentence.py)
– Hybrid Scoring Module (conbine_ranking_score.py)
– BM25 Retriever (via search.py)
– Final Hit Aggregation & Filtering
D. Reranking
– BGE Reranker model invoked post-retrieval
– Optional LLM re-ranking paths
E. Model Training Subsystems (“training”)
– retrieval/: training scripts for E5, GTE, BGE-M3
– rerank/: training BGE reranker
– llm/: scripts & code for SFT & GRPO fine-tuning, plus prompt/reward modules
– DeepRetrieval/ & ViSearch-R1/: RL-based query rewriting & self-searching training servers
F. Evaluation (“eval”)
– Scripts that invoke retrieval pipeline, compute metrics
Relationships & Data Flows
• Pipeline queries DB & index → returns candidate docs → reranker → final results
• Training subsystems read from prepared corpora & ground-truth labels → update model artifacts (checkpoint outputs)
• Eval jobs call pipeline components & produce metric outputs
Architectural Patterns & Principles
• Modular, stage-based pipeline (Separation of Concerns)
• Pluggable retrieval heads (allow E5, GTE, BM25, LLM modules to be added/removed)
• Config-driven behavior (config/*.json for DeepSpeed & PPO settings)
• Scalability via external vector DB (Qdrant) and shell-script orchestration
Technologies & Frameworks
• Python services & scripts
• Qdrant (vector storage)
• Hugging Face transformers for embeddings & LLM
• BM25 (e.g., via Whoosh or Elastic/Pyserini)
• DeepSpeed ZeRO for large-model training
• RL libraries for PPO/GRPO in LLM training
Final Checklist
[ ] All main components represented
[ ] Data flows and dependencies are directional and unambiguous
[ ] Architectural patterns (modular stages, config-driven) are annotated
[ ] External services (Qdrant, Hugging Face APIs) are shown
[ ] Optional/future LLM blocks are clearly marked