# hieutran2019/text_retrieval architecture

> Architecture diagram of the GitHub repository hieutran2019/text_retrieval, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hieutran2019/text_retrieval
- Repository: https://github.com/hieutran2019/text_retrieval

## Overview

##### Project type and purpose
- This is a data-science/NLP pipeline project implemented in a Jupyter Notebook (“Text_Retrieval.ipynb”).
- Its purpose is to perform text retrieval on the MS MARCO test dataset: load queries and passages, index or embed the passages, retrieve and rank relevant passages for each query, and evaluate the results.

##### File structure
- Top level contains only:
- README.md (project description, data schema, pipeline overview)
- Text_Retrieval.ipynb (the actual code implementing ingestion, retrieval, ranking, evaluation)
- No separate “src” or “lib” directories—everything is orchestrated in a single notebook.
- Dependencies (implied) include Hugging Face Datasets, tokenizers/ML frameworks (e.g., PyTorch or TensorFlow), retrieval/ranking libraries (e.g., FAISS, BM25).

##### README for additional insights
- Data source: MS MARCO v1.1 test split via Hugging Face Datasets.
- No external services or microservices; it’s a stand-alone pipeline script.

##### Detailed breakdown of components
- MS MARCO v1.1 test split (via HF Datasets)
2. Ingestion Module
- HF Datasets loader → Python objects
3. Preprocessing Module
- Text cleaning → Tokenization → Feature extraction
4. Indexing Component
- Build term index (BM25) or vector index (FAISS)
5. Query Processing
- Input query → Tokenize → Query vector or term list
6. Retrieval & Scoring
- Run query against index → Score passages
7. Ranking & Output
- Sort by score → Select top-k → Format results
8. Evaluation
- Use ground-truth labels (answers) → Compute MRR/MAP
9. Notebook Orchestration
- Each stage is a separate notebook cell or modular function

## Diagram (Mermaid)

```mermaid
graph TB
  subgraph "Jupyter Notebook Environment"
    direction TB
    PO["Project Overview & Pipeline Description\n(README.md)"]:::note
    DS[("MS MARCO v1.1 test split\n(via HF Datasets)")]:::db
    ING["Ingestion Module\n(HF Datasets loader)"]:::data
    PRE["Preprocessing Module\n(text cleaning & tokenization)"]:::data
    IDX["Indexing Component\n(BM25/FAISS)"]:::data
    QP["Query Processing\n(tokenization/embedding)"]:::retrieve
    RS["Retrieval & Scoring\n(run queries & score)"]:::retrieve
    RO["Ranking & Output\n(sort & format top-k)"]:::retrieve
    EV["Evaluation Module\n(compute MRR/MAP)"]:::eval
  end

  PO --- DS
  DS --> ING --> PRE --> IDX
  QP --> IDX
  IDX --> RS
  QP --> RS
  RS --> RO --> EV

  click PO "https://github.com/hieutran2019/text_retrieval/blob/main/README.md"
  click DS "https://github.com/hieutran2019/text_retrieval/blob/main/Text_Retrieval.ipynb"
  click ING "https://github.com/hieutran2019/text_retrieval/blob/main/Text_Retrieval.ipynb"
  click PRE "https://github.com/hieutran2019/text_retrieval/blob/main/Text_Retrieval.ipynb"
  click IDX "https://github.com/hieutran2019/text_retrieval/blob/main/Text_Retrieval.ipynb"
  click QP "https://github.com/hieutran2019/text_retrieval/blob/main/Text_Retrieval.ipynb"
  click RS "https://github.com/hieutran2019/text_retrieval/blob/main/Text_Retrieval.ipynb"
  click RO "https://github.com/hieutran2019/text_retrieval/blob/main/Text_Retrieval.ipynb"
  click EV "https://github.com/hieutran2019/text_retrieval/blob/main/Text_Retrieval.ipynb"

  classDef db fill:#cccccc,stroke:#333,stroke-width:2px;
  classDef data fill:#ADD8E6,stroke:#333,stroke-width:1px;
  classDef retrieve fill:#90EE90,stroke:#333,stroke-width:1px;
  classDef eval fill:#FFA500,stroke:#333,stroke-width:1px;
  classDef note fill:#f0f0f0,stroke:#666,stroke-width:1px,stroke-dasharray: 4 2;
```
