# luongxuannhat/learning-to-rank-with-rank-net-algorithm architecture

> Architecture diagram of the GitHub repository luongxuannhat/learning-to-rank-with-rank-net-algorithm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/luongxuannhat/learning-to-rank-with-rank-net-algorithm
- Repository: https://github.com/luongxuannhat/learning-to-rank-with-rank-net-algorithm

## Overview

##### Project Type and Purpose
- This is a standalone data‐processing + machine‐learning pipeline (not a web service or microservice architecture).
- Purpose: fetch Vietnamese proverbs (“ca dao, tục ngữ”), preprocess them with TF-IDF, generate training pairs, train a RankNet model (PyTorch), then score and rank documents.

##### File Structure
- FetchData/: data‐fetching component (crawler script + raw JSON outputs + stopwords).
- Library/TF_IDF/: custom TF-IDF feature‐extractor module.
- Library/rank_net.py: RankNet model definition (PyTorch).
- helper.py: utility functions (load JSON, remove stopwords, compute relevance scores).
- main.py: orchestrator that ties all phases (preprocess → feature extraction → pair generation → model training → prediction & ranking).
- Config/data files at root: vietnamese-stopwords.txt, training_pairs.txt

##### README for Architectural Insights
- Six logical stages: data‐fetching, preprocessing, TF-IDF representation, relevance‐scoring, training‐pair generation, RankNet training & ranking.
- No external microservices; all in‐process Python modules.

##### Main Components & Their Relationships
a. Data Acquisition
- Component: FetchData/crawl_data.py
- Inputs: web pages (via chromedriver)
- Outputs: FetchData/output/*.json
b. Data Preprocessing
- Component: helper.py
- Inputs: JSON documents + vietnamese-stopwords.txt
- Outputs: tokenized documents
c. Feature Extraction
- Component: Library/TF_IDF/TF_IDF.py
- Inputs: tokenized docs
- Outputs: TF-IDF matrix + vocabulary
d. Training-Pair Generation
- Component: helper.py or main.py logic
- Inputs: TF-IDF vectors + a sample query
- Outputs: pairs (i,j) with labels → training_pairs.txt (or in-memory)
e. Model Training
- Component: Library/rank_net.py + PyTorch
- Inputs: training pairs + features
- Outputs: trained RankNet model
f. Prediction & Ranking
- Component: main.py
- Inputs: trained model + TF-IDF features + query
- Outputs: sorted list of documents with scores

##### Architectural Patterns & Design Principles
- Pipeline pattern: clearly separated sequential stages.
- Modularization: each concern lives in its own module/directory.
- Single‐entry orchestrator: main.py invokes each stage.

##### Essential Details to Capture
- Management of stopwords as an external file.
- Generation and use of training pairs.
- Flow of feature vectors into model.
- Separation between feature‐building and model logic.

## Diagram (Mermaid)

```mermaid
flowchart TD
    DataExist{"Data exists? (JSON)"}:::decision
    Crawler["Data Acquisition\nFetchData/crawl_data.py"]:::process
    RawJSON1["cadao_tucngu_complete.json"]:::dataStore
    RawJSON2["cadao_tucngu_mini.json"]:::dataStore
    Stopwords["vietnamese-stopwords.txt"]:::dataStore
    Preprocessor["Data Preprocessing\nhelper.py"]:::process
    Tokenized["Tokenized Documents"]:::dataStore
    TFIDF["Feature Extraction\nLibrary/TF_IDF/TF_IDF.py"]:::process
    TFIDFMatrix["TF-IDF Matrix & Vocabulary"]:::dataStore
    PairGen["Training-Pair Generation\nhelper.py"]:::process
    TrainingPairsFile["training_pairs.txt"]:::dataStore
    RankNet["Model Training\nLibrary/rank_net.py"]:::process
    TrainedModel["Trained RankNet Model"]:::dataStore
    Predictor["Prediction & Ranking\nmain.py"]:::process
    RankedOutput["Ranked Documents List"]:::dataStore

    DataExist --No--> Crawler
    DataExist --Yes--> RawJSON1
    Crawler --> RawJSON1
    Crawler --> RawJSON2
    RawJSON1 --> Preprocessor
    RawJSON2 --> Preprocessor
    Stopwords --> Preprocessor
    Preprocessor --> Tokenized
    Tokenized --> TFIDF
    TFIDF --> TFIDFMatrix
    TFIDFMatrix --> PairGen
    PairGen --> TrainingPairsFile
    TrainingPairsFile --> RankNet
    TFIDFMatrix --> RankNet
    RankNet --> TrainedModel
    TrainedModel --> Predictor
    TFIDFMatrix --> Predictor
    Predictor --> RankedOutput

    click Crawler "https://github.com/luongxuannhat/learning-to-rank-with-rank-net-algorithm/blob/master/FetchData/crawl_data.py"
    click RawJSON1 "https://github.com/luongxuannhat/learning-to-rank-with-rank-net-algorithm/blob/master/FetchData/output/cadao_tucngu_complete.json"
    click RawJSON2 "https://github.com/luongxuannhat/learning-to-rank-with-rank-net-algorithm/blob/master/FetchData/output/cadao_tucngu_mini.json"
    click Stopwords "https://github.com/luongxuannhat/learning-to-rank-with-rank-net-algorithm/blob/master/vietnamese-stopwords.txt"
    click Preprocessor "https://github.com/luongxuannhat/learning-to-rank-with-rank-net-algorithm/blob/master/helper.py"
    click TFIDF "https://github.com/luongxuannhat/learning-to-rank-with-rank-net-algorithm/blob/master/Library/TF_IDF/TF_IDF.py"
    click PairGen "https://github.com/luongxuannhat/learning-to-rank-with-rank-net-algorithm/blob/master/helper.py"
    click PairGen "https://github.com/luongxuannhat/learning-to-rank-with-rank-net-algorithm/blob/master/main.py"
    click TrainingPairsFile "https://github.com/luongxuannhat/learning-to-rank-with-rank-net-algorithm/blob/master/training_pairs.txt"
    click RankNet "https://github.com/luongxuannhat/learning-to-rank-with-rank-net-algorithm/blob/master/Library/rank_net.py"
    click Predictor "https://github.com/luongxuannhat/learning-to-rank-with-rank-net-algorithm/blob/master/main.py"

    classDef dataStore fill:#ADD8E6,stroke:#000,stroke-width:1px,shape:cylinder
    classDef process fill:#90EE90,stroke:#000,stroke-width:1px
    classDef orchestrator fill:#FFA500,stroke:#000,stroke-width:1px
    classDef decision fill:#fff2cc,stroke:#000,stroke-width:1px,shape:diamond
```
