# vlm-run/vlmrun-cookbook architecture

> Architecture diagram of the GitHub repository vlm-run/vlmrun-cookbook, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vlm-run/vlmrun-cookbook
- Repository: https://github.com/vlm-run/vlmrun-cookbook

## Overview

##### Project type and purpose
- This repo is not a monolithic application but a “cookbook” of ready-to-run Jupyter/Colab notebooks demonstrating how to interact with the VLM Run Platform.
- Its goal is educational: show users how to extract structured data from images, video, and documents using Vision-Language Models (VLMs) via the VLM Run API/SDK.

##### File structure
- Top-level files (.gitignore, Makefile, .pre-commit-config.yaml) support repo maintenance and CI.
- assets/financial-presentations contains sample input data.
- There is no “src/” folder—this repo is pure examples, not the platform’s codebase.

##### README for additional insights
- The README confirms that each notebook is a self-contained Colab example calling the VLM Run API.
- It lists use cases (financial docs, drivers’ licenses, TV news, etc.) and highlights features like schema generation, visual grounding, video transcription.

##### Essential architectural elements to capture
– The separation between example code (notebooks) and the production platform.
– The end-to-end call chain: Notebook → SDK → API → Inference → Notebook.
– The modular microservice layout, which enables new features (schemas, video, advanced finetuning) to be added seamlessly.
– Storage and metadata layers that support reproducibility and scale.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Subgraph: Local Development Environment
    subgraph "Local Development Environment" 
        direction TB
        Notebooks["Jupyter/Colab Notebooks"]:::frontend
        Quickstart["00_quickstart.ipynb"]:::frontend
        SchemaShowcase["01_schema_showcase.ipynb"]:::frontend
        DriversLicense["02_case_study_drivers_license.ipynb"]:::frontend
        TVNews["03_case_study_tv_news.ipynb"]:::frontend
        VisualGrounding["04_visual_grounding.ipynb"]:::frontend
        ImageCatalogue["05_case_study_image_catalogue.ipynb"]:::frontend
        FashionSearch["06_fashion_images_hybrid_search.ipynb"]:::frontend
        SchemaGen["07_generate_schema.ipynb"]:::frontend
        AdvFinetuneVideo["advanced_finetuning_video_inference.ipynb"]:::frontend
        AdvVideoTrans["advanced_video_transcription.ipynb"]:::frontend
        Assets["Sample Assets\n(assets/financial-presentations/)"]:::frontend
        README["README.md"]:::frontend
        CIConfig[".gitignore\n.pre-commit-config.yaml\nMakefile"]:::frontend
    end

    %% Subgraph: Cloud/API Layer
    subgraph "Cloud/API" 
        direction TB
        SDK["VLM Run SDK\n(Python Client)"]:::api
        APIGateway["API Gateway\n(REST/HTTP)"]:::api
    end

    %% Subgraph: VLM Run Backend
    subgraph "VLM Run Backend" 
        direction TB
        Ingestion["Ingestion & Preprocessing Service"]:::backend
        SchemaEngine["Schema Engine Service"]:::backend
        Inference["Inference Service\n(VLM Models)"]:::backend
    end

    %% Subgraph: Storage & Models
    subgraph "Storage & Models"
        direction TB
        ObjectStore["Object Store\n(e.g., S3)"]:::storage
        MetadataStore["Metadata Store\n(DynamoDB/PostgreSQL)"]:::storage
        ModelCluster["Model Cluster\n(GPU Instances)"]:::storage
    end

    %% Optional UI
    WebUI["VLM Run Web UI (Optional)"]:::api

    %% Connections
    Notebooks -->|imports SDK & auth| SDK
    SDK -->|Send HTTP Request| APIGateway
    APIGateway -->|Route & Fetch Assets| Ingestion
    Ingestion -->|Validate Schema| SchemaEngine
    SchemaEngine -->|Invoke Model| Inference
    Inference -->|Store Outputs| ObjectStore
    Inference -->|Log Metadata| MetadataStore
    Inference -->|Return JSON| APIGateway
    APIGateway -->|Return Response| SDK
    SDK -->|Render Results| Notebooks
    WebUI -->|Also calls SDK| SDK

    %% Click Events
    click Notebooks "https://github.com/vlm-run/vlmrun-cookbook/tree/main/notebooks/"
    click Quickstart "https://github.com/vlm-run/vlmrun-cookbook/blob/main/notebooks/00_quickstart.ipynb"
    click SchemaShowcase "https://github.com/vlm-run/vlmrun-cookbook/blob/main/notebooks/01_schema_showcase.ipynb"
    click DriversLicense "https://github.com/vlm-run/vlmrun-cookbook/blob/main/notebooks/02_case_study_drivers_license.ipynb"
    click TVNews "https://github.com/vlm-run/vlmrun-cookbook/blob/main/notebooks/03_case_study_tv_news.ipynb"
    click VisualGrounding "https://github.com/vlm-run/vlmrun-cookbook/blob/main/notebooks/04_visual_grounding.ipynb"
    click ImageCatalogue "https://github.com/vlm-run/vlmrun-cookbook/blob/main/notebooks/05_case_study_image_catalogue.ipynb"
    click FashionSearch "https://github.com/vlm-run/vlmrun-cookbook/blob/main/notebooks/06_fashion_images_hybrid_search.ipynb"
    click SchemaGen "https://github.com/vlm-run/vlmrun-cookbook/blob/main/notebooks/07_generate_schema.ipynb"
    click AdvFinetuneVideo "https://github.com/vlm-run/vlmrun-cookbook/blob/main/notebooks/advanced_finetuning_video_inference.ipynb"
    click AdvVideoTrans "https://github.com/vlm-run/vlmrun-cookbook/blob/main/notebooks/advanced_video_transcription.ipynb"
    click Assets "https://github.com/vlm-run/vlmrun-cookbook/tree/main/assets/financial-presentations/"
    click README "https://github.com/vlm-run/vlmrun-cookbook/blob/main/README.md"
    click CIConfig "https://github.com/vlm-run/vlmrun-cookbook/blob/main/.gitignore"
    click CIConfig "https://github.com/vlm-run/vlmrun-cookbook/blob/main/.pre-commit-config.yaml"
    click CIConfig "https://github.com/vlm-run/vlmrun-cookbook/tree/main/Makefile"

    %% Styles
    classDef frontend fill:#D0E6FF,stroke:#0033CC,color:#000000;
    classDef api fill:#D0F7D0,stroke:#2E8B57,color:#000000;
    classDef backend fill:#DEF1FF,stroke:#1E90FF,color:#000000;
    classDef storage fill:#FFF2CC,stroke:#DAA520,color:#000000;
```
