How hieutran2019/text_retrieval works
Project type and purpose
• This is a data-science/NLP pipeline project implemented in a Jupyter Notebook (“Text_Retrieval.ipynb”).
• Its purpose is to perform text retrieval on the MS MARCO test dataset: load queries and passages, index or embed the passages, retrieve and rank relevant passages for each query, and evaluate the results.
File structure
• Top level contains only:
• README.md (project description, data schema, pipeline overview)
• Text_Retrieval.ipynb (the actual code implementing ingestion, retrieval, ranking, evaluation)
• No separate “src” or “lib” directories—everything is orchestrated in a single notebook.
• Dependencies (implied) include Hugging Face Datasets, tokenizers/ML frameworks (e.g., PyTorch or TensorFlow), retrieval/ranking libraries (e.g., FAISS, BM25).
README for additional insights
• Data source: MS MARCO v1.1 test split via Hugging Face Datasets.
• No external services or microservices; it’s a stand-alone pipeline script.
Detailed breakdown of components
• MS MARCO v1.1 test split (via HF Datasets)
2. Ingestion Module
• HF Datasets loader → Python objects
3. Preprocessing Module
• Text cleaning → Tokenization → Feature extraction
4. Indexing Component
• Build term index (BM25) or vector index (FAISS)
5. Query Processing
• Input query → Tokenize → Query vector or term list
6. Retrieval & Scoring
• Run query against index → Score passages
7. Ranking & Output
• Sort by score → Select top-k → Format results
8. Evaluation
• Use ground-truth labels (answers) → Compute MRR/MAP
9. Notebook Orchestration
• Each stage is a separate notebook cell or modular function