# kasys-lab/pr-rank architecture

> Architecture diagram of the GitHub repository kasys-lab/pr-rank, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kasys-lab/pr-rank
- Repository: https://github.com/kasys-lab/pr-rank

## Overview

##### Project Type & Purpose
- This project is a modular experimental system for processing and ranking data from large text datasets (e.g., AOL-IA and DMOZ) using learning-to-rank methods.
- It involves multiple experimental phases: generating query relevance judgments (qrel), extracting features from queries/documents, dividing datasets for experiments, and performing parameter regression to train a ranking model.
- The purpose is to experiment with different feature sets (e.g., Q, D, Q-D) and parameters to improve ranking performance.

##### Main Components
a. External Resources:
- Data resources (AOL-IA dataset, DMOZ dataset).
- FastText model (for language detection).
- RankLib (for training the learning-to-rank model).

b. Modular Components (each represented as a distinct block):
- AOL-Qrel Module:
- Handles tasks like language detection (using FastText) and initial query/document filtering.
- Contains configuration (config/config.yaml), data folders, and a main.py execution script.
- Features Extraction Module:
- Responsible for preprocessing documents and queries, and extracting various features (query, document, and query-document).
- Uses spacy (as indicated by the download command for en_core_web_sm) and contains its own configuration.
- Dataset Division Module:
- Divides the dataset using clustering methods and partitions the data for subsequent learning-to-rank experiments.
- Configurable via its own config.yaml which allows choosing specific query features, etc.
- Parameter Regression Module:
- Trains and evaluates the learning-to-rank (PR-Rank) model via parameter regression.
- This module also has its dedicated configuration that controls feature selection and experiment naming.
- Utilities:
- Common shared utilities (e.g., text, file, ranklib, and other helper functions).
- Experiment Management:
- Although implicit, the experiment naming conventions and configuration files indicate that experiments are run in sequence, so you may include a “Coordinator” or “Experiment Runner” element that orchestrates execution.

##### Relationships and Interactions
- From there, the output (e.g., filtered and annotated dataset) serves as input for the Features Extraction module.
- Next, features feed into the Dataset Division module where the dataset is clustered and divided, potentially generating intermediate representations.
- Finally, the processed data is used in the Parameter Regression module to train and evaluate the ranking models.
- Additionally, indicate that configuration files (each module’s config.yaml) guide the behavior of their respective modules.

##### Architectural Patterns & Design Principles
- Modularity: Each experimental phase is contained in its own subfolder with a clear separation of concerns.
- Config-driven behavior: Each module is controlled by its config file, allowing for experiment-specific settings.
- Pipeline-like processing: Data flows sequentially from pre-processing (qrel, features extraction) to dataset splitting, and finally to model training.
- Use of external libraries: Incorporate dependencies such as FastText, spaCy, and RankLib.

## Diagram (Mermaid)

```mermaid
graph TD
  %% External Resources
  subgraph "External Resources"
    AOLDataset["AOL-IA dataset"]:::ellipse
    DMOZdataset["DMOZ dataset"]:::ellipse
    FastText["FastText model"]:::ellipse
    RankLib["RankLib"]:::ellipse
  end

  %% Modules
  subgraph "Modules"
    AOLQ["AOL-Qrel Module\n(config/config.yaml, main.py)"]:::module
    FE["Features Extraction Module\n(config/config.yaml, main.py, src scripts)"]:::module
    DD["Dataset Division Module\n(config/config.yaml, main.py, clustering, division)"]:::module
    PR["Parameter Regression Module\n(config/config.yaml, main.py, training, evaluating)"]:::module
  end

  %% Other components
  UTIL["Utilities\n(aolia_utils.py, file_utils.py, ranklib_utils.py, text_utils.py)"]:::utils
  EXP["Experiment Runner"]:::runner

  %% Data Flow
  AOLDataset -->|"feeds"| AOLQ
  DMOZdataset -->|"feeds"| AOLQ
  FastText -->|"languageDetection"| AOLQ
  AOLQ -->|"filteredData"| FE
  FE -->|"extractedFeatures"| DD
  DD -->|"dividedData"| PR
  PR -->|"trainedModels"| EXP
  RankLib -->|"modelTraining"| PR

  %% Utilities usage by all modules
  AOLQ ---|"uses"| UTIL
  FE ---|"uses"| UTIL
  DD ---|"uses"| UTIL
  PR ---|"uses"| UTIL

  %% Click Events
  click AOLQ "https://github.com/kasys-lab/pr-rank/tree/main/PR-Rank/aolia_qrel"
  click FE "https://github.com/kasys-lab/pr-rank/tree/main/PR-Rank/features_extraction"
  click DD "https://github.com/kasys-lab/pr-rank/tree/main/PR-Rank/dataset_division"
  click PR "https://github.com/kasys-lab/pr-rank/tree/main/PR-Rank/parameter_regression"
  click UTIL "https://github.com/kasys-lab/pr-rank/tree/main/PR-Rank/utils"

  %% Styles
  classDef ellipse fill:#ffdead,stroke:#000,stroke-width:2px;
  classDef module fill:#add8e6,stroke:#000,stroke-width:2px;
  classDef utils fill:#90ee90,stroke:#000,stroke-width:2px;
  classDef runner fill:#d3d3d3,stroke:#000,stroke-width:2px;
```
