# casia-iva-lab/anomalygpt architecture

> Architecture diagram of the GitHub repository casia-iva-lab/anomalygpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/casia-iva-lab/anomalygpt
- Repository: https://github.com/casia-iva-lab/anomalygpt

## Overview

##### Project Type and Purpose
- AnomalyGPT is a full-stack research-oriented application focused on industrial anomaly detection using a combination of visual and language models.
- According to the README, it leverages a pre-trained image encoder (ImageBind) and a large language model (Vicuna/PandaGPT) to both detect and provide descriptive details about image anomalies.
- Its purpose is to streamline anomaly detection without manual thresholding by aligning image features with textual cues.

##### Analyzing the File Structure
- Top-Level Directories:
- “code”: Contains the core source code. It is subdivided into configuration (code/config, code/dsconfig), datasets (code/datasets), models (code/model), training scripts (code/train*.py, code/scripts), utilities (code/utils), and a web demo (code/web_demo.py).
- “pretrained_ckpt”: Contains subdirectories for pre-trained checkpoints for both ImageBind and Vicuna/PandaGPT.
- “data”: Houses all training and evaluation data, including anomaly datasets.
- “images”: Contains logos and demo images used in the README.
- Architectural Indicators:
- The separation of components into modules (datasets, config, model, utils) strongly hints at a modular, layered approach.
- The existence of shell scripts and dedicated training/test files indicates a clear separation between training/inference and configuration.

##### Main Components
a. Data Layer:
- Data Sources – Represented by the “data” directory that contains datasets like MVTec-AD, VisA, and PandaGPT visual instruction data.
- Data Preprocessing – Within “code/datasets” and “code/utils/data_transform.py” where image sampling, self-supervised tasks, and dataset loaders are implemented.
b. Configurations:
- Configuration Manager – “code/config” (YAML files) and “code/dsconfig” (JSON config for training) which define hyperparameters, training modes, and pipeline settings.
c. Model Components:
- Image Encoder – The ImageBind module (located at code/model/ImageBind) that extracts visual features.
- Language Model Adapter – The integration with models such as Vicuna (and pretrained PandaGPT weights). Files like “code/model/modeling_llama.py”, “code/model/openllama.py” and “code/model/agent.py” inform the multi-modal alignment.
- Anomaly Detection Mechanism – The anomaly detection logic is primarily found across “code/model/AnomalyGPT_models.py” and additional processing in “code/utils”.
d. Training Pipeline:
- Training Scripts – Scripts under “code/scripts” and commands triggered via “code/train_mvtec.py”, “code/train_visa.py” which orchestrate the training loop.
- Loss, Logging & Registry – Modules under “code/utils/logger.py”, “code/utils/loss.py”, “code/utils/registry.py” manage training monitoring and configurations.
e. Inference and Demo:
- Web Demo – “code/web_demo.py” which exposes the application for testing the anomaly detection in a user-facing interface.
f. Pre-trained Checkpoints:
- “pretrained_ckpt” structure indicates that checkpoints for different models (ImageBind, PandaGPT, Vicuna) are loaded at runtime, emphasizing modularity.

##### Relationships and Flow
- Data Flow:
- Raw input data (industrial images and textual prompts) is ingested from the “data” layer, then preprocessed by dataset components.
- The preprocessed data feeds into the training scripts that use configuration files from “code/config” and “code/dsconfig”.
- Model Interaction:
- The processed images are passed through the Image Encoder (ImageBind module) to extract features.
- These features, alongside prompts or anomaly descriptions, interact with the language model (adapted via PandaGPT’s delta weights and Vicuna) to generate anomaly indications.
- The outputs include anomaly scores and localization info provided by the lightweight visual-textual feature matching.
- Deployment Flow:
- Once trained, models with updated delta weights (provided or trained) are loaded during inference via the “web_demo.py”, which handles input/output interactions.
- Utility Connections:
- Various utilities (logging, I/O, helper functions) underpin the training, model building, and configuration management, ensuring smooth data flow and operational logging.

##### Architectural Patterns and Technologies
- Modular and Layered Architecture – Separation into data, configurations, models, utilities, and interfaces.
- Integration of Pre-trained Components – Reusing external models (ImageBind, Vicuna, PandaGPT) and their respective checkpoints.
- Config Driven – Deployment and training parameters are largely managed through YAML and JSON configuration files.
- Use of Shell Scripting & Command Line Interface – For reproducible training/inference execution.

##### Final Advice for the Principal Software Engineer
- Be detailed and exhaustive; every core module should appear as a distinct component.
- Leverage the provided file tree as a guide to identify all logical blocks (e.g., embedding configuration, dataset management, model integration, training flows, and deployment).
- Remember to include both the “backend” model training/inference pipeline and a “frontend” component for the demo interface (web_demo.py).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        Data1["Data Sources (data)"]:::data
        Data2["Datasets (code/datasets)"]:::data
        DP["Data Preprocessing (code/utils/data_transform.py)"]:::data
    end

    Data1 -->|"provides"| Data2
    Data2 -->|"feeds_data"| DP

    %% Configuration Manager
    subgraph "Configuration Manager"
        YAML["YAML Configurations (code/config)"]:::config
        JSON["JSON Configurations (code/dsconfig)"]:::config
    end

    %% Training Pipeline
    subgraph "Training Pipeline"
        TS["Training Scripts (code/train_mvtec.py)"]:::training
        TU["Training Utilities (code/utils/logger.py)"]:::training
    end

    TU -->|"supports_TS"| TS

    %% Model Components
    subgraph "Model Components"
        ImgEnc["Image Encoder (ImageBind)"]:::model
        LM["Language Model Adapter"]:::model
        AD["Anomaly Detection Mechanism"]:::model
    end

    %% Inference and Demo
    subgraph "Inference & Demo"
        Demo["Web Demo Interface (code/web_demo.py)"]:::demo
    end

    %% Pre-trained Checkpoints
    subgraph "Pre-trained Checkpoints"
        PT["Pre-trained Checkpoints"]:::checkpoint
    end

    %% Data flow connections
    DP -->|"feeds"| TS
    YAML -->|"configures"| TS
    JSON -->|"configures"| TS

    TS -->|"trains"| ImgEnc
    TS -->|"trains"| LM
    TS -->|"trains"| AD

    %% Additional configuration flow to model components
    YAML -->|"affects"| ImgEnc
    JSON -->|"affects"| AD

    %% Pre-trained checkpoints feeding model components
    PT -->|"loads_ckpt"| ImgEnc
    PT -->|"loads_ckpt"| LM

    %% Model output to demo
    ImgEnc -->|"outputs"| Demo
    LM -->|"outputs"| Demo
    AD -->|"outputs"| Demo

    %% Click Events
    click Data1 "https://github.com/casia-iva-lab/anomalygpt/tree/main/data"
    click Data2 "https://github.com/casia-iva-lab/anomalygpt/tree/main/code/datasets"
    click DP "https://github.com/casia-iva-lab/anomalygpt/blob/main/code/utils/data_transform.py"
    click YAML "https://github.com/casia-iva-lab/anomalygpt/tree/main/code/config"
    click JSON "https://github.com/casia-iva-lab/anomalygpt/tree/main/code/dsconfig"
    click ImgEnc "https://github.com/casia-iva-lab/anomalygpt/tree/main/code/model/ImageBind"
    click LM "https://github.com/casia-iva-lab/anomalygpt/blob/main/code/model/modeling_llama.py"
    click AD "https://github.com/casia-iva-lab/anomalygpt/blob/main/code/model/AnomalyGPT_models.py"
    click TS "https://github.com/casia-iva-lab/anomalygpt/blob/main/code/train_mvtec.py"
    click TU "https://github.com/casia-iva-lab/anomalygpt/blob/main/code/utils/logger.py"
    click Demo "https://github.com/casia-iva-lab/anomalygpt/blob/main/code/web_demo.py"
    click PT "https://github.com/casia-iva-lab/anomalygpt/tree/main/pretrained_ckpt"

    %% Styles
    classDef data fill:#cce5ff,stroke:#333,stroke-width:2px;
    classDef config fill:#d4edda,stroke:#333,stroke-width:2px;
    classDef training fill:#fef9c3,stroke:#333,stroke-width:2px;
    classDef model fill:#f8d7da,stroke:#333,stroke-width:2px;
    classDef demo fill:#e2e3e5,stroke:#333,stroke-width:2px;
    classDef checkpoint fill:#d1ecf1,stroke:#333,stroke-width:2px;
```
