# bfshi/absvit architecture

> Architecture diagram of the GitHub repository bfshi/absvit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bfshi/absvit
- Repository: https://github.com/bfshi/absvit

## Overview

##### Project type and purpose
- This is a research‐oriented, multi‐task computer‐vision codebase built around Vision Transformers (ViTs) and top‐down attention (AbSViT).
- It supports three main pipelines:
- Image Classification (main.py, engine.py, models/, datasets.py)
- Semantic Segmentation (segmentation/…)
- Vision‐Language Tasks (vision_language/, with the “meter” library)

##### File structure
- Root level
- main.py: CLI entry for classification training/evaluation
- engine.py: training/eval loop, optimizer, scheduler, metrics
- config.py, datasets.py: classification configuration and data loaders
- models/: ViT and AbSViT model definitions
- requirements.txt: global dependencies
- segmentation/
- configs/: config‐driven segmentation experiments (top‐down, bottom‐up ViT)
- models/: segmentation head implementations
- segmentation/README.md: usage instructions
- vision_language/
- run.py: CLI entry for V–L tasks
- setup.py, meter/: packaged “meter” library
– datamodules/: PyTorch Lightning–style data pipelines for VQA, captioning, etc.
– datasets/, transforms/: preprocessing and dataset wrappers
– modules/: core V–L model components (ViT, BERT, CLIP, heads, objectives)
– utils/, gadgets/: helper functions and metrics
- demo/: Jupyter notebooks for attention visualization

##### Key insights from README
- The code is highly modular and config‐driven
- Each subfolder (classification, segmentation, V–L) has its own entrypoint, configs, and model definitions
- External services: none (all on‐prem), but uses external datasets and pre‐trained weights

##### Final reminder
- Essential components (data loaders, configs, model definitions, training loops, outputs)
- How each task reuses the ViT core but extends it with task‐specific heads and configs
- Keep it detailed enough for someone unfamiliar with AbSViT to understand the end‐to‐end pipelines for classification, segmentation, and vision‐language tasks

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% Classification Pipeline
  subgraph "Classification Pipeline"
    direction TB
    CLI_C["Classification CLI Entrypoint"]:::blue
    CFG_C["Classification Config Manager"]:::blue
    DATA_C["Classification Data Ingestion"]:::blue
    subgraph "Classification Models"
      direction TB
      AB["AbSViT Core"]:::blue
      VT["ViT Core"]:::blue
    end
    ENGINE["Training & Evaluation Engine"]:::shared
    CLI_C -->|load configs| CFG_C
    CFG_C -->|select dataset| DATA_C
    DATA_C -->|data→| AB
    DATA_C -->|data→| VT
    AB -->|model→| ENGINE
    VT -->|model→| ENGINE
    CFG_C -->|params→| ENGINE
  end

  %% Segmentation Pipeline
  subgraph "Segmentation Pipeline"
    direction TB
    CLI_S["Segmentation Usage & Entrypoint"]:::green
    subgraph "Segmentation Config Manager"
      direction TB
      SUBCFG1["vit_bottom_up configs"]:::green
      SUBCFG2["vit_top_down configs"]:::green
    end
    DATA_S["Segmentation Data & Transforms"]:::green
    subgraph "Segmentation Models"
      direction TB
      SBU["Bottom-Up Head"]:::green
      STD["Top-Down Head"]:::green
    end
    CLI_S -->|read README| SUBCFG1
    CLI_S -->|read README| SUBCFG2
    SUBCFG1 -->|select dataset| DATA_S
    SUBCFG2 -->|select dataset| DATA_S
    DATA_S -->|data→| SBU
    DATA_S -->|data→| STD
    SBU -->|model→| ENGINE
    STD -->|model→| ENGINE
    SUBCFG1 -->|params→| ENGINE
    SUBCFG2 -->|params→| ENGINE
  end

  %% Vision-Language Pipeline
  subgraph "Vision-Language Pipeline"
    direction TB
    CLI_V["V-L CLI Entrypoint"]:::orange
    CFG_V["V-L Config Manager"]:::orange
    DM["V-L Data Ingestion"]:::orange
    DS["V-L Datasets & Preprocessing"]:::orange
    TR["V-L Transforms & Augmentations"]:::orange
    VM["V-L Model Definitions"]:::orange
    UTIL["V-L Utilities & Metrics"]:::orange
    CLI_V -->|load configs| CFG_V
    CFG_V -->|select datamodule| DM
    DM -->|datamodules→| DS
    DS -->|preprocessed→| TR
    TR -->|augmented→| VM
    VM -->|model→| ENGINE
    CFG_V -->|params→| ENGINE
  end

  %% Shared Core
  subgraph "Shared Components"
    direction TB
    CORE["ViT Backbone Core"]:::shared
    ENGINE
    CORE -->|reuse| AB
    CORE -->|reuse| VT
    CORE -->|reuse| SBU
    CORE -->|reuse| STD
    CORE -->|reuse| VM
  end

  %% Outputs
  Outputs["Outputs & Artifacts"]:::artifact
  ENGINE -->|"save→"| Outputs

  %% Click Events
  click CLI_C "https://github.com/bfshi/absvit/blob/master/main.py"
  click CFG_C "https://github.com/bfshi/absvit/blob/master/config.py"
  click DATA_C "https://github.com/bfshi/absvit/blob/master/datasets.py"
  click AB "https://github.com/bfshi/absvit/blob/master/models/absvit.py"
  click VT "https://github.com/bfshi/absvit/blob/master/models/vit.py"
  click CLI_S "https://github.com/bfshi/absvit/blob/master/segmentation/README.md"
  click SUBCFG1 "https://github.com/bfshi/absvit/tree/master/segmentation/configs/vit_bottom_up/"
  click SUBCFG2 "https://github.com/bfshi/absvit/tree/master/segmentation/configs/vit_top_down/"
  click SBU "https://github.com/bfshi/absvit/blob/master/segmentation/models/vit_bottom_up.py"
  click STD "https://github.com/bfshi/absvit/blob/master/segmentation/models/vit_top_down.py"
  click CLI_V "https://github.com/bfshi/absvit/blob/master/vision_language/run.py"
  click CFG_V "https://github.com/bfshi/absvit/blob/master/vision_language/meter/config.py"
  click DM "https://github.com/bfshi/absvit/tree/master/vision_language/meter/datamodules/"
  click DS "https://github.com/bfshi/absvit/tree/master/vision_language/meter/datasets/"
  click VM "https://github.com/bfshi/absvit/tree/master/vision_language/meter/modules/"
  click TR "https://github.com/bfshi/absvit/tree/master/vision_language/meter/transforms/"
  click UTIL "https://github.com/bfshi/absvit/tree/master/vision_language/meter/utils/"
  click UTIL "https://github.com/bfshi/absvit/tree/master/vision_language/meter/gadgets/"

  %% Styles
  classDef blue fill:#E3F2FD,stroke:#1976D2,color:#0D47A1;
  classDef green fill:#E8F5E9,stroke:#388E3C,color:#1B5E20;
  classDef orange fill:#FFF3E0,stroke:#FB8C00,color:#E65100;
  classDef shared fill:#F3E5F5,stroke:#8E24AA,color:#4A148C;
  classDef artifact fill:#ECEFF1,stroke:#546E7A,color:#263238;
```
