How luongxuannhat/learning-to-rank-with-rank-net-algorithm works
Project Type and Purpose
• This is a standalone data‐processing + machine‐learning pipeline (not a web service or microservice architecture).
• Purpose: fetch Vietnamese proverbs (“ca dao, tục ngữ”), preprocess them with TF-IDF, generate training pairs, train a RankNet model (PyTorch), then score and rank documents.
File Structure
• FetchData/: data‐fetching component (crawler script + raw JSON outputs + stopwords).
• Library/TF_IDF/: custom TF-IDF feature‐extractor module.
• Library/rank_net.py: RankNet model definition (PyTorch).
• helper.py: utility functions (load JSON, remove stopwords, compute relevance scores).
• main.py: orchestrator that ties all phases (preprocess → feature extraction → pair generation → model training → prediction & ranking).
• Config/data files at root: vietnamese-stopwords.txt, training_pairs.txt
README for Architectural Insights
• Six logical stages: data‐fetching, preprocessing, TF-IDF representation, relevance‐scoring, training‐pair generation, RankNet training & ranking.
• No external microservices; all in‐process Python modules.
Main Components & Their Relationships
a. Data Acquisition
• Component: FetchData/crawl_data.py
• Inputs: web pages (via chromedriver)
• Outputs: FetchData/output/*.json
b. Data Preprocessing
• Component: helper.py
• Inputs: JSON documents + vietnamese-stopwords.txt
• Outputs: tokenized documents
c. Feature Extraction
• Component: Library/TF_IDF/TF_IDF.py
• Inputs: tokenized docs
• Outputs: TF-IDF matrix + vocabulary
d. Training-Pair Generation
• Component: helper.py or main.py logic
• Inputs: TF-IDF vectors + a sample query
• Outputs: pairs (i,j) with labels → training_pairs.txt (or in-memory)
e. Model Training
• Component: Library/rank_net.py + PyTorch
• Inputs: training pairs + features
• Outputs: trained RankNet model
f. Prediction & Ranking
• Component: main.py
• Inputs: trained model + TF-IDF features + query
• Outputs: sorted list of documents with scores
Architectural Patterns & Design Principles
• Pipeline pattern: clearly separated sequential stages.
• Modularization: each concern lives in its own module/directory.
• Single‐entry orchestrator: main.py invokes each stage.
Essential Details to Capture
• Management of stopwords as an external file.
• Generation and use of training pairs.
• Flow of feature vectors into model.
• Separation between feature‐building and model logic.