# keio-smilab24/polos architecture

> Architecture diagram of the GitHub repository keio-smilab24/polos, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/keio-smilab24/polos
- Repository: https://github.com/keio-smilab24/polos

## Overview

##### Main Components
- Library Core and CLI: The core functionality of Polos is encapsulated in the “polos” package. Here, modules like cli.py expose the library’s interface for command-line interactions.
- Metrics: The “polos/metrics” folder holds modules for computing both ranking and regression metrics. This design indicates that after feature extraction and estimation, results are evaluated by comparing models’ predictions to human judgements.
- Tokenizers: The “polos/tokenizers_” directory contains different tokenizer implementations conforming to the underlying text processing needs (e.g., Hugging Face tokenizers for XLMR).
- Supporting Modules: Directories like “polos/modules” (with activations, feedforward, and scalar mix functions) and “polos/schedulers.py” support the neural architecture’s inner workings.
- Trainer and Evaluation: The trainer.py script and train.sh indicate a training pipeline, while the validate folder and scripts like validate.sh manage the evaluation process. These components control model training, checkpoint handling, and performance assessment.
- Configuration and Installation: Files like configs/polos-trainer.yaml and install.sh indicate configurable training parameters and the necessary installation procedures, respectively.

##### Flow and Relationships
- Data Input & Preprocessing: Illustrate a flow where image and text data (possibly coming from datasets like Polaris or Flickr8K – as specified in the README) are first processed. The image input is handled (and possibly preprocessed via CLIP-based modules) while text data runs through tokenizers.
- Feature Extraction: Show parallel feature extraction paths where one branch might use CLIP (or related techniques) and another leverages text encoders (e.g., BERT or XLMR found under “polos/models/encoders”). Both branches contribute to generating a multimodal feature representation.
- Estimation & Scoring: The extracted features then feed into estimation modules (located in “polos/models/estimators”) to predict evaluation scores. An additional ranking branch further refines these outcomes.
- Metric Calculation: Post-estimation, the metrics modules (in “polos/metrics”) calculate performance scores that are compared with human judgements.

##### Architectural Patterns and Key Technologies
- Modular Design: The clear separation into directories (models, tokenizers, metrics, modules) is indicative of a highly modular, plug-and-play design which enhances maintainability and extensibility.
- Parallel Feature Extraction: The system is carefully designed to process multimodal inputs in parallel (using CLIP and text feature extractors) before merging the results.
- Pipeline Flow: The project embodies one of the standard ML application architectures with distinct stages for data ingestion, feature processing, model training, and evaluation.
- External Dependencies: Notable external libraries include CLIP, PyTorch (with Lightning for training), and tokenizers from Hugging Face. CUDA support is also a critical part of the training/prediction components.
- Configuration Management: Use of configuration files in “configs” allows parameter tuning and experiment configurability.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input & Preprocessing Layer
    subgraph "Input & Preprocessing"
        RawImage["Raw Image Data"]:::external
        RawText["Raw Text Data"]:::external
        subgraph "Tokenizers"
            HFTokenizer["HF Tokenizer"]:::preprocess
            XLMRTokenizer["XLMR Tokenizer"]:::preprocess
        end
        CLIP["CLIP Image Processor"]:::preprocess
        RawText --> HFTokenizer
        RawText --> XLMRTokenizer
        RawImage --> CLIP
    end

    %% Core Computation Layer
    subgraph "Core Computation"
        CLI["CLI Interface"]:::core
        subgraph "Text Encoders"
            BERT["BERT Encoder"]:::core
            XLMR["XLMR Encoder"]:::core
        end
        subgraph "Estimators"
            PolosEst["Polos Estimator"]:::core
            QualityEst["Quality Estimator"]:::core
        end
        Ranking["Polos Ranker"]:::core
        subgraph "Metrics Computation"
            RankingMetrics["Ranking Metrics"]:::core
            RegressionMetrics["Regression Metrics"]:::core
        end
        subgraph "Supporting Modules"
            Activations["Activations"]:::core
            Feedforward["Feedforward"]:::core
            ScalarMix["Scalar Mix"]:::core
        end
        Schedulers["Schedulers"]:::core
    end

    %% Training & Evaluation Layer
    subgraph "Training & Evaluation"
        Trainer["Trainer"]:::training
        TrainScript["Train Script"]:::training
        subgraph "Validation"
            ClipScore["Clip Score"]:::training
            ComputeCorr["Compute Correlations"]:::training
        end
        ValidateScript["Validation Script"]:::training
        Config["Trainer Config"]:::training
        Installation["Installation Script"]:::training
    end

    %% Data Flow Connections
    HFTokenizer -->|"feeds"| BERT
    XLMRTokenizer -->|"feeds"| XLMR
    BERT -->|"extracts features"| PolosEst
    XLMR -->|"extracts features"| QualityEst
    CLIP -->|"extracts features"| PolosEst
    CLIP -->|"extracts features"| QualityEst
    Activations -->|"supports"| PolosEst
    Feedforward -->|"supports"| PolosEst
    ScalarMix -->|"supports"| PolosEst
    PolosEst -->|"outputs"| Ranking
    QualityEst -->|"outputs"| Ranking
    Ranking -->|"calculates"| RankingMetrics
    Ranking -->|"calculates"| RegressionMetrics
    CLI -->|"triggers"| Trainer
    Ranking -->|"supplies input"| Trainer
    Schedulers -->|"schedules"| Trainer
    Trainer -->|"executes"| TrainScript
    Trainer -->|"validates"| ClipScore
    Trainer -->|"validates"| ComputeCorr
    Trainer -->|"runs"| ValidateScript
    Config -->|"configures"| Trainer
    Installation -->|"sets up"| CLI

    %% Click Events
    click CLI "https://github.com/keio-smilab24/polos/blob/master/polos/cli.py"
    click BERT "https://github.com/keio-smilab24/polos/blob/master/polos/models/encoders/bert.py"
    click XLMR "https://github.com/keio-smilab24/polos/blob/master/polos/models/encoders/xlmr.py"
    click PolosEst "https://github.com/keio-smilab24/polos/blob/master/polos/models/estimators/polos_estimator.py"
    click QualityEst "https://github.com/keio-smilab24/polos/blob/master/polos/models/estimators/quality_estimator.py"
    click Ranking "https://github.com/keio-smilab24/polos/blob/master/polos/models/ranking/polos_ranker.py"
    click RankingMetrics "https://github.com/keio-smilab24/polos/blob/master/polos/metrics/ranking_metrics.py"
    click RegressionMetrics "https://github.com/keio-smilab24/polos/blob/master/polos/metrics/regression_metrics.py"
    click HFTokenizer "https://github.com/keio-smilab24/polos/blob/master/polos/tokenizers_/hf_tokenizer.py"
    click XLMRTokenizer "https://github.com/keio-smilab24/polos/blob/master/polos/tokenizers_/xlmr_tokenizer.py"
    click Activations "https://github.com/keio-smilab24/polos/blob/master/polos/modules/activations.py"
    click Feedforward "https://github.com/keio-smilab24/polos/blob/master/polos/modules/feedforward.py"
    click ScalarMix "https://github.com/keio-smilab24/polos/blob/master/polos/modules/scalar_mix.py"
    click Schedulers "https://github.com/keio-smilab24/polos/blob/master/polos/schedulers.py"
    click Trainer "https://github.com/keio-smilab24/polos/blob/master/polos/trainer.py"
    click TrainScript "https://github.com/keio-smilab24/polos/blob/master/train.sh"
    click ClipScore "https://github.com/keio-smilab24/polos/blob/master/validate/clip_score.py"
    click ComputeCorr "https://github.com/keio-smilab24/polos/blob/master/validate/compute_correlations.py"
    click ValidateScript "https://github.com/keio-smilab24/polos/blob/master/validate.sh"
    click Config "https://github.com/keio-smilab24/polos/blob/master/configs/polos-trainer.yaml"
    click Installation "https://github.com/keio-smilab24/polos/blob/master/install.sh"

    %% Styles
    classDef external fill:#CEF6EC,stroke:#2E8B57,stroke-width:2px;
    classDef preprocess fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef core fill:#FAD7A0,stroke:#E67E22,stroke-width:2px;
    classDef training fill:#D2B4DE,stroke:#8E44AD,stroke-width:2px;
```
