How keio-smilab24/polos works
Main Components
• Library Core and CLI: The core functionality of Polos is encapsulated in the “polos” package. Here, modules like cli.py expose the library’s interface for command-line interactions.
• Metrics: The “polos/metrics” folder holds modules for computing both ranking and regression metrics. This design indicates that after feature extraction and estimation, results are evaluated by comparing models’ predictions to human judgements.
• Tokenizers: The “polos/tokenizers_” directory contains different tokenizer implementations conforming to the underlying text processing needs (e.g., Hugging Face tokenizers for XLMR).
• Supporting Modules: Directories like “polos/modules” (with activations, feedforward, and scalar mix functions) and “polos/schedulers.py” support the neural architecture’s inner workings.
• Trainer and Evaluation: The trainer.py script and train.sh indicate a training pipeline, while the validate folder and scripts like validate.sh manage the evaluation process. These components control model training, checkpoint handling, and performance assessment.
• Configuration and Installation: Files like configs/polos-trainer.yaml and install.sh indicate configurable training parameters and the necessary installation procedures, respectively.
Flow and Relationships
• Data Input & Preprocessing: Illustrate a flow where image and text data (possibly coming from datasets like Polaris or Flickr8K – as specified in the README) are first processed. The image input is handled (and possibly preprocessed via CLIP-based modules) while text data runs through tokenizers.
• Feature Extraction: Show parallel feature extraction paths where one branch might use CLIP (or related techniques) and another leverages text encoders (e.g., BERT or XLMR found under “polos/models/encoders”). Both branches contribute to generating a multimodal feature representation.
• Estimation & Scoring: The extracted features then feed into estimation modules (located in “polos/models/estimators”) to predict evaluation scores. An additional ranking branch further refines these outcomes.
• Metric Calculation: Post-estimation, the metrics modules (in “polos/metrics”) calculate performance scores that are compared with human judgements.
Architectural Patterns and Key Technologies
• Modular Design: The clear separation into directories (models, tokenizers, metrics, modules) is indicative of a highly modular, plug-and-play design which enhances maintainability and extensibility.
• Parallel Feature Extraction: The system is carefully designed to process multimodal inputs in parallel (using CLIP and text feature extractors) before merging the results.
• Pipeline Flow: The project embodies one of the standard ML application architectures with distinct stages for data ingestion, feature processing, model training, and evaluation.
• External Dependencies: Notable external libraries include CLIP, PyTorch (with Lightning for training), and tokenizers from Hugging Face. CUDA support is also a critical part of the training/prediction components.
• Configuration Management: Use of configuration files in “configs” allows parameter tuning and experiment configurability.