How microsoft/geo-vision-labeler works
Project Type and Purpose
• GeoVision Labeler is an open-source, command-line, geospatial image labeling tool.
• It ingests satellite or aerial images, uses a vision LLM to generate detailed image descriptions, applies one of several classification strategies (OpenAI API, Hugging Face LLM, or CLIP), and outputs labeled metadata.
• It also provides a separate clustering pipeline to organize user-provided classes into hierarchical meta-classes, and a visualization tool to overlay labels on images.
File Structure Analysis
Top level:
• AUTHORS.rst, CODE_OF_CONDUCT.md, LICENSE, SECURITY.md, README.md – project metadata
• environment.yml, setup.py – environment and packaging
• data/ – static input/output data
– classes.txt (flat classes), labels.csv (output), spacenet_eval.csv (benchmark data)
• src/ – source code modules:
– main.py – orchestrator for the labeling pipeline
– cluster.py – hierarchical class clustering script
– vision.py – image description module
– classifier.py – LLM classification module
– clip.py – CLIP classifier fallback/standalone
– display_chunks.py – visualization script
– __init__.py
README Insights
• Describes a modular design with interchangeable description and classification components.
• Highlights three classification modes: ‘openai’, ‘huggingface’, ‘clip’.
• Explains hierarchical clustering to produce YAML meta-classes used in multi-level classification.
• Outlines command-line interface options and external dependencies (Hugging Face API, OpenAI/Azure).
Core Components
a. User/CLI Layer
• main.py (Labeling Pipeline)
• cluster.py (Class Clustering)
• display_chunks.py (Visualization)
b. Processing Modules
• Image Chunking & Preprocessing
• vision.py (Vision LLM prompt & inference)
• classifier.py (LLM text-classification)
• clip.py (CLIP fallback classifier)
• CSV/YAML Writers & Readers
c. Data Stores
• Local filesystem: raw images, classes.txt, generated clusters.yaml, labels.csv, evaluation CSV
d. External Services
• Hugging Face Inference API (vision & HF LLM models)
• OpenAI/Azure OpenAI API (LLM classification)
e. Environment & Packaging
• Conda environment (environment.yml)
• Python packaging (setup.py)
Interactions & Data Flow
1. User invokes python src/main.py …
2. main.py reads input_dir & classes_file
3. Image Chunking → batch of patches
4. Each patch → vision.py → vision LLM API → textual description
5. Description + (optional) classes list & filename → classifier.py → chosen LLM API
– If no valid label → clip.py → CLIP API → fallback label
6. Aggregated metadata → writer → labels.csv
7. (Optional) User invokes python src/cluster.py … to generate clusters.yaml
8. (Optional) User invokes python src/display_chunks.py … to overlay labels on image
Architectural Patterns & Principles
• Modular, plugin-style components for description and classification
• CLI orchestration separating ingestion, processing, and output
• Fail-over strategy: primary LLM classifier → CLIP fallback
• Recursive, zero-shot clustering pipeline for hierarchical classification