How bfshi/absvit works
Project type and purpose
• This is a research‐oriented, multi‐task computer‐vision codebase built around Vision Transformers (ViTs) and top‐down attention (AbSViT).
• It supports three main pipelines:
• Image Classification (main.py, engine.py, models/, datasets.py)
• Semantic Segmentation (segmentation/…)
• Vision‐Language Tasks (vision_language/, with the “meter” library)
File structure
• Root level
• main.py: CLI entry for classification training/evaluation
• engine.py: training/eval loop, optimizer, scheduler, metrics
• config.py, datasets.py: classification configuration and data loaders
• models/: ViT and AbSViT model definitions
• requirements.txt: global dependencies
• segmentation/
• configs/: config‐driven segmentation experiments (top‐down, bottom‐up ViT)
• models/: segmentation head implementations
• segmentation/README.md: usage instructions
• vision_language/
• run.py: CLI entry for V–L tasks
• setup.py, meter/: packaged “meter” library
– datamodules/: PyTorch Lightning–style data pipelines for VQA, captioning, etc.
– datasets/, transforms/: preprocessing and dataset wrappers
– modules/: core V–L model components (ViT, BERT, CLIP, heads, objectives)
– utils/, gadgets/: helper functions and metrics
• demo/: Jupyter notebooks for attention visualization
Key insights from README
• The code is highly modular and config‐driven
• Each subfolder (classification, segmentation, V–L) has its own entrypoint, configs, and model definitions
• External services: none (all on‐prem), but uses external datasets and pre‐trained weights
Final reminder
• Essential components (data loaders, configs, model definitions, training loops, outputs)
• How each task reuses the ViT core but extends it with task‐specific heads and configs
• Keep it detailed enough for someone unfamiliar with AbSViT to understand the end‐to‐end pipelines for classification, segmentation, and vision‐language tasks