# lllyasviel/omost architecture

> Architecture diagram of the GitHub repository lllyasviel/omost, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lllyasviel/omost
- Repository: https://github.com/lllyasviel/omost

## Overview

##### Project Type and Purpose
- Omost is an open-source Python tool that leverages LLMs (Llama3/Phi3 variants) to generate code for composing images via a virtual Canvas API, then renders those compositions through a diffusion-based image generator. It is neither a monolithic web app nor a pure library; it sits between an LLM chat interface (CLI/Gradio) and a diffusion renderer, forming a middleware pipeline.

##### File Structure Analysis
- Top-level scripts:
- chat_interface.py – a simple CLI or chat wrapper for conversational editing.
- gradio_app.py – a Gradio-based web UI that accepts text prompts and displays images.
- lib_omost/
- canvas.py – defines the Canvas class with set_global_description and add_local_description.
- pipeline.py – orchestrates LLM calls, memory management, merging sub-prompts, and handing off to the renderer.
- memory_management.py – tracks conversation history, prompt bags, and context windows.
- requirements.txt – Torch, diffusers, bitsandbytes, huggingface-hub, Gradio, etc.

##### Key Architectural Insights from README
- LLMs produce strictly structured “Canvas” calls, not raw image tokens.
- A “Region Prompter” (attention manipulation) or external control model performs region-guided diffusion.
- A two-stage flow: (1) LLM → Canvas instructions; (2) Canvas instructions → latent map → diffusion render.

##### Main Components
a. Frontend Layer
- CLI (chat_interface.py) and Gradio UI (gradio_app.py)
- HTTP/Web Socket (Gradio) or stdin/stdout (CLI)
b. Core Library Layer
- Memory Manager – maintains prompt history, sub-prompt bags, context lengths
- Pipeline –
– Sends prompt + history to LLM API (local or Hugging Face inference)
– Receives Python code that invokes Canvas methods
– Parses/validates Canvas calls → internal data structures
– Assembles merged sub-prompt bags
c. Rendering Layer
- Region Prompter / Attention Manipulation Module (baseline renderer in repo)
- Diffusion Model (e.g., Stable Diffusion via diffusers library)
- Denoising Scheduler
d. External Services & Dependencies
- LLM Models (omost-llama-3-8b, omost-phi-3-mini-128k, quant versions) via Hugging Face or local
- GPU (CUDA/cu121) and bitsandbytes for quant inference
- Gradio for hosting the web UI
- Optional: external control nets (Gligen, InstanceDiffusion)

##### Relationships and Data Flow
1. User → CLI/Gradio UI (text prompt)
2. UI → Pipeline + Memory Manager → construct LLM payload
3. Pipeline → LLM Model → returns Canvas API code
4. Pipeline → execute/parse Canvas calls → Canvas data structure
6. Region Prompter → manipulate attention masks/scores per region
7. Diffusion Model → generate image latent → decode → final image
8. Image → UI → User

##### Architectural Patterns & Principles
- Pipeline Pattern: clear sequencing of Text → Code → DataModel → Render
- Separation of Concerns: UI vs. LLM orchestration vs. Canvas logic vs. renderer
- Plugin-like LLM Backend: interchangeable model providers (Hugging Face, local)
- Data-Driven Composition: Canvas calls as schema-driven DSL for image composition

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% UI Layer
    subgraph "UI Layer"
        CLI["CLI"]:::ui
        Gradio["Gradio UI"]:::ui
    end

    %% Core Library Layer
    subgraph "Core Library"
        Memory[(Memory Manager)]:::core
        Pipeline["Pipeline Orchestrator"]:::core
        Canvas["Canvas Agent"]:::core
    end

    %% Rendering Layer
    subgraph "Rendering Layer"
        Region["Region Prompter"]:::renderer
        Latent["Latent Initialization"]:::renderer
        Diffuse["Diffusion Model"]:::renderer
        Denoise["Denoising Scheduler"]:::renderer
    end

    %% External Services
    subgraph "External Services"
        LLM["LLM Models"]:::external
        GPU["GPU & bitsandbytes"]:::external
        ControlNet["External Control Nets"]:::external
        Requirements["requirements.txt"]:::external
    end

    %% Data Flow
    CLI -->|"prompt"| Pipeline
    Gradio -->|"prompt"| Pipeline

    Pipeline -->|"history query"| Memory
    Memory -->|"context"| Pipeline

    Pipeline -->|"LLM payload"| LLM
    LLM -->|"Canvas code"| Pipeline

    Pipeline -->|"Canvas data"| Canvas
    Canvas -->|"merged sub-prompts"| Region
    Pipeline -->|"depth map + color"| Latent

    Region -->|"attention masks"| Latent
    Latent -->|"initialized latent"| Diffuse
    Diffuse -->|"denoised latent"| Denoise
    Denoise -->|"image"| CLI
    Denoise -->|"image"| Gradio

    %% Optional Integrations
    Pipeline -.->|"optional control"| ControlNet
    Pipeline -.->|"quant inference"| GPU

    %% Click Events
    click CLI "https://github.com/lllyasviel/omost/blob/main/chat_interface.py"
    click Gradio "https://github.com/lllyasviel/omost/blob/main/gradio_app.py"
    click Memory "https://github.com/lllyasviel/omost/blob/main/lib_omost/memory_management.py"
    click Pipeline "https://github.com/lllyasviel/omost/blob/main/lib_omost/pipeline.py"
    click Canvas "https://github.com/lllyasviel/omost/blob/main/lib_omost/canvas.py"
    click Requirements "https://github.com/lllyasviel/omost/blob/main/requirements.txt"

    %% Styles
    classDef ui fill:#a2f5a2,stroke:#333,stroke-width:1px
    classDef core fill:#a2d2f5,stroke:#333,stroke-width:1px
    classDef renderer fill:#d9a2f5,stroke:#333,stroke-width:1px
    classDef external fill:#dddddd,stroke:#333,stroke-width:1px
```
