# microsoft/geo-vision-labeler architecture

> Architecture diagram of the GitHub repository microsoft/geo-vision-labeler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/geo-vision-labeler
- Repository: https://github.com/microsoft/geo-vision-labeler

## Overview

##### Project Type and Purpose
- GeoVision Labeler is an open-source, command-line, geospatial image labeling tool.
- It ingests satellite or aerial images, uses a vision LLM to generate detailed image descriptions, applies one of several classification strategies (OpenAI API, Hugging Face LLM, or CLIP), and outputs labeled metadata.
- It also provides a separate clustering pipeline to organize user-provided classes into hierarchical meta-classes, and a visualization tool to overlay labels on images.

##### File Structure Analysis
Top level:
- AUTHORS.rst, CODE_OF_CONDUCT.md, LICENSE, SECURITY.md, README.md – project metadata
- environment.yml, setup.py – environment and packaging
- data/ – static input/output data
– classes.txt (flat classes), labels.csv (output), spacenet_eval.csv (benchmark data)
- src/ – source code modules:
– main.py – orchestrator for the labeling pipeline
– cluster.py – hierarchical class clustering script
– vision.py – image description module
– classifier.py – LLM classification module
– clip.py – CLIP classifier fallback/standalone
– display_chunks.py – visualization script
– __init__.py

##### README Insights
- Describes a modular design with interchangeable description and classification components.
- Highlights three classification modes: ‘openai’, ‘huggingface’, ‘clip’.
- Explains hierarchical clustering to produce YAML meta-classes used in multi-level classification.
- Outlines command-line interface options and external dependencies (Hugging Face API, OpenAI/Azure).

##### Core Components
a. User/CLI Layer
- main.py (Labeling Pipeline)
- cluster.py (Class Clustering)
- display_chunks.py (Visualization)
b. Processing Modules
- Image Chunking & Preprocessing
- vision.py (Vision LLM prompt & inference)
- classifier.py (LLM text-classification)
- clip.py (CLIP fallback classifier)
- CSV/YAML Writers & Readers
c. Data Stores
- Local filesystem: raw images, classes.txt, generated clusters.yaml, labels.csv, evaluation CSV
d. External Services
- Hugging Face Inference API (vision & HF LLM models)
- OpenAI/Azure OpenAI API (LLM classification)
e. Environment & Packaging
- Conda environment (environment.yml)
- Python packaging (setup.py)

##### Interactions & Data Flow
1. User invokes `python src/main.py …`
2. main.py reads input_dir & classes_file
3. Image Chunking → batch of patches
4. Each patch → vision.py → vision LLM API → textual description
5. Description + (optional) classes list & filename → classifier.py → chosen LLM API
– If no valid label → clip.py → CLIP API → fallback label
6. Aggregated metadata → writer → labels.csv
7. (Optional) User invokes `python src/cluster.py …` to generate clusters.yaml
8. (Optional) User invokes `python src/display_chunks.py …` to overlay labels on image

##### Architectural Patterns & Principles
- Modular, plugin-style components for description and classification
- CLI orchestration separating ingestion, processing, and output
- Fail-over strategy: primary LLM classifier → CLIP fallback
- Recursive, zero-shot clustering pipeline for hierarchical classification

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        main["src/main.py"]:::core
        cluster["src/cluster.py"]:::core
        display["src/display_chunks.py"]:::core
    end

    %% Processing Layer
    subgraph "Processing Layer"
        chunker["Image Chunker"]:::core
        vision["src/vision.py"]:::core
        classifier["src/classifier.py"]:::core
        clip["src/clip.py"]:::core
        writer["CSV/YAML Writer"]:::core
    end

    %% Data Stores
    subgraph "Data Stores"
        classes["data/classes.txt"]:::data
        clusters["clusters.yaml"]:::data
        labels["data/labels.csv"]:::data
        bench["data/spacenet_eval.csv"]:::data
        images["Raw Images"]:::data
    end

    %% External Services
    subgraph "External Services"
        hf["Hugging Face API"]:::ext
        openai["OpenAI/Azure API"]:::ext
    end

    %% Environment & Packaging
    subgraph "Environment & Packaging"
        env["environment.yml"]:::env
        pkg["setup.py"]:::env
    end

    %% Data Flows
    main -->|"reads classes.txt"| classes
    main -->|"reads raw images"| images
    main -->|"triggers chunking"| chunker
    chunker -->|"image patch"| vision
    vision -->|"LLM request"| hf
    hf -->|"description"| vision
    vision -->|"description"| classifier
    classifier -->|"label request"| openai
    openai -->|"label"| classifier
    classifier -->|"if no label fallback"| clip
    clip -->|"label"| classifier
    classifier -->|"metadata batch"| writer
    writer -->|"writes labels.csv"| labels

    cluster -->|"reads classes.txt"| classes
    cluster -->|"generates meta-classes"| clusters
    clusters -->|"used by main (hierarchical)"| main

    display -->|"reads labels.csv"| labels
    display -->|"reads raw images"| images
    display -->|"outputs overlays"| images

    %% Click Events
    click main "https://github.com/microsoft/geo-vision-labeler/blob/main/src/main.py"
    click cluster "https://github.com/microsoft/geo-vision-labeler/blob/main/src/cluster.py"
    click display "https://github.com/microsoft/geo-vision-labeler/blob/main/src/display_chunks.py"
    click vision "https://github.com/microsoft/geo-vision-labeler/blob/main/src/vision.py"
    click classifier "https://github.com/microsoft/geo-vision-labeler/blob/main/src/classifier.py"
    click clip "https://github.com/microsoft/geo-vision-labeler/blob/main/src/clip.py"
    click classes "https://github.com/microsoft/geo-vision-labeler/blob/main/data/classes.txt"
    click labels "https://github.com/microsoft/geo-vision-labeler/blob/main/data/labels.csv"
    click bench "https://github.com/microsoft/geo-vision-labeler/blob/main/data/spacenet_eval.csv"
    click env "https://github.com/microsoft/geo-vision-labeler/blob/main/environment.yml"
    click pkg "https://github.com/microsoft/geo-vision-labeler/blob/main/setup.py"

    %% Styles
    classDef core fill:#cce5ff,stroke:#004085
    classDef data fill:#d4edda,stroke:#155724
    classDef ext fill:#fff3cd,stroke:#856404
    classDef env fill:#f5c6cb,stroke:#721c24
```
