How doem97/deblora works
Purpose and execution surface
This repository implements experiments for “De-Biased Representation Learning for Long-tailed PEFT in Remote Sensing.” Its workflow compares zero-shot CLIP, full fine-tuning, LoRA, and calibrated/pLoRA-style representations, then evaluates them through feature extraction, clustering/calibration, and linear probing. Primary Python entry points are main.py and trainer.py; reproducible experiment sequences are exposed as shell launchers under exp/, such as exp/extract_lora_feat.sh and exp/lora_linprob.sh.
Configuration-driven experiment selection
Experiments are composed from dataset, backbone, and tuner YAMLs rather than hardcoded per-dataset programs. Dataset definitions cover long-tailed natural-image and remote-sensing datasets; model definitions select CLIP, ImageNet-21K ViT, or SatMAE variants; tuner definitions select Flora/LoRA parameter-efficient adaptation variants.
• configs/data/dota.yaml
• configs/model/clip_vit_b16.yaml
• configs/tuner/flora.yaml
Dataset ingestion and long-tail protocol
Dataset adapters provide class labels and train/validation/test splits for CIFAR-100, DOTA, FUSRS, ImageNet-LT, iNaturalist 2018, and Places-LT. Shared long-tailed data handling is centralized separately from individual dataset loaders. Dataset metadata and split files are repository assets, while image/data roots are expected under data/ or linked there.
• datasets/lt_data.py
• datasets/dota.py
• datasets/imagenet_lt.py
Backbone and adaptation layer
The model layer wraps visual encoders and classifiers, including CLIP visual/text processing, SatMAE ViTs, and PEFT modules. The vendored clip/ package supplies CLIP model construction and tokenization; project wrappers in models/ integrate these encoders into training/probing workflows. A second src/models/ branch supports DIFT/diffusion-derived feature backbones and alternate CLIP/DINO implementations.
• models/models.py
• models/peft_modules.py
• clip/model.py
Feature production and persistence boundary
Feature extraction is a first-class intermediate stage: zero-shot, fine-tuned, and LoRA representations are extracted and written for later reuse. Precomputed feature artifacts can be downloaded through Hugging Face-oriented scripts, decoupling expensive encoder inference from downstream clustering and probes. Diffusion/intermediate feature extraction scripts form a parallel path for remote-sensing experiments.
• exp/extract_0shot_feat.sh
• output/features/hf_features.sh
• scripts/extract_feat_mid+lora.py
Calibration, clustering, and evaluation flow
The pLoRA workflow consumes LoRA features, clusters feature space, calibrates representations, and then runs linear probes. Clustering supports CPU/GPU K-means and GPU HDBSCAN, indicating optional RAPIDS/CUDA infrastructure; rapids.hdbscan.doc documents that dependency boundary. Classifier/probe implementations are separated from model adaptation code.
• scripts/feat_cluster_kmeans_gpu_fp32_v0.3.py
• scripts/feat_cluster_hdbscan_gpu_fp32_v0.4.py
• models/classifiers.py
Runtime and external dependencies
The project is Python 3.8-oriented, uses PyTorch-style model training/inference implied by the CLIP and ViT implementations, and installs dependencies from requirements.txt. External services/assets include Hugging Face-hosted datasets/features and locally provisioned image datasets. GPU execution is material for LoRA training, feature extraction, and GPU clustering; CPU clustering alternatives are also present.
• requirements.txt
• data/hf_dataset.sh
• scripts/feat_cluster_cpu_kmeans_v0.2.py
Main entry points: main.py Main entry point · trainer.py Training orchestration · hf_features.sh Downloaded features · models.py Encoder wrappers
Components 21
Experiment Configuration
Composable dataset, backbone, and tuner selection.
- Dataset config
configs/data/dota.yaml - Backbone config
configs/model/clip_vit_b16.yaml - Tuner config
configs/tuner/flora.yaml
Data and Assets
Long-tailed datasets and reusable feature artifacts.
- Dataset adapters · Provide labels and train, validation, and test splits.
datasets/dota.py - Long-tail protocol · Shared long-tailed sampling and split handling.
datasets/lt_data.py - Local image datasets · Image roots are provisioned or linked locally.
data/hf_dataset.sh - Downloaded features · Precomputed artifacts decouple inference from downstream analysis.
output/features/hf_features.sh
Models and Adaptation
Encoders, CLIP, and PEFT adaptation.
- Encoder wrappers · Integrates visual encoders into training and probing.
models/models.py - LoRA / Flora modules
models/peft_modules.py - Vendored CLIP · Constructs CLIP models and tokenization.
clip/model.py - Probe classifiers
models/classifiers.py
Experiment Pipeline
Training, extraction, calibration, and probing.
- Experiment launchers · Reproducible sequences for extraction and linear probing.
exp/extract_lora_feat.sh - Main entry point
main.py - Training orchestration · Runs full fine-tuning and PEFT experiments.
trainer.py - Feature extraction · Produces zero-shot, fine-tuned, and LoRA representations.
exp/extract_0shot_feat.sh - Diffusion feature path · Extracts intermediate diffusion-derived remote-sensing features.
scripts/extract_feat_mid+lora.py - GPU clustering · Optional CUDA/RAPIDS clustering for calibration.
scripts/feat_cluster_kmeans_gpu_fp32_v0.3.py - CPU clustering · CPU alternative to GPU clustering.
scripts/feat_cluster_cpu_kmeans_v0.2.py - Calibrated representations · pLoRA-style representations produced from clustered LoRA features.
- Linear probing · Evaluates extracted and calibrated representations.
exp/lora_linprob.sh
Other components
- Python dependencies · Python 3.8 and PyTorch-oriented runtime dependencies.
requirements.txt
Connections 21
- Experiment launchers → Main entry point: starts
- Main entry point → Training orchestration: orchestrates
- Dataset config → Main entry point: selects data
- Backbone config → Main entry point: selects backbone
- Tuner config → Main entry point: selects adaptation
- Training orchestration → Long-tail protocol: uses
- Long-tail protocol → Dataset adapters: coordinates splits
- Dataset adapters → Local image datasets: loads
- Training orchestration → Encoder wrappers: trains
- Encoder wrappers → Vendored CLIP: wraps
- Encoder wrappers → LoRA / Flora modules: applies
- Training orchestration → Feature extraction: produces models for
- Feature extraction → Downloaded features: writes or reuses
- Diffusion feature path → Downloaded features: adds alternate features
- Downloaded features → GPU clustering: clusters
- Downloaded features → CPU clustering: clusters
- GPU clustering → Calibrated representations: calibrates
- CPU clustering → Calibrated representations: calibrates
- Calibrated representations → Linear probing: evaluates
- Linear probing → Probe classifiers: uses
- Python dependencies → Main entry point: runtime