How lllyasviel/omost works
Project Type and Purpose
• Omost is an open-source Python tool that leverages LLMs (Llama3/Phi3 variants) to generate code for composing images via a virtual Canvas API, then renders those compositions through a diffusion-based image generator. It is neither a monolithic web app nor a pure library; it sits between an LLM chat interface (CLI/Gradio) and a diffusion renderer, forming a middleware pipeline.
File Structure Analysis
• Top-level scripts:
• chat_interface.py – a simple CLI or chat wrapper for conversational editing.
• gradio_app.py – a Gradio-based web UI that accepts text prompts and displays images.
• lib_omost/
• canvas.py – defines the Canvas class with set_global_description and add_local_description.
• pipeline.py – orchestrates LLM calls, memory management, merging sub-prompts, and handing off to the renderer.
• memory_management.py – tracks conversation history, prompt bags, and context windows.
• requirements.txt – Torch, diffusers, bitsandbytes, huggingface-hub, Gradio, etc.
Key Architectural Insights from README
• LLMs produce strictly structured “Canvas” calls, not raw image tokens.
• A “Region Prompter” (attention manipulation) or external control model performs region-guided diffusion.
• A two-stage flow: (1) LLM → Canvas instructions; (2) Canvas instructions → latent map → diffusion render.
Main Components
a. Frontend Layer
• CLI (chat_interface.py) and Gradio UI (gradio_app.py)
• HTTP/Web Socket (Gradio) or stdin/stdout (CLI)
b. Core Library Layer
• Memory Manager – maintains prompt history, sub-prompt bags, context lengths
• Pipeline –
– Sends prompt + history to LLM API (local or Hugging Face inference)
– Receives Python code that invokes Canvas methods
– Parses/validates Canvas calls → internal data structures
– Assembles merged sub-prompt bags
c. Rendering Layer
• Region Prompter / Attention Manipulation Module (baseline renderer in repo)
• Diffusion Model (e.g., Stable Diffusion via diffusers library)
• Denoising Scheduler
d. External Services & Dependencies
• LLM Models (omost-llama-3-8b, omost-phi-3-mini-128k, quant versions) via Hugging Face or local
• GPU (CUDA/cu121) and bitsandbytes for quant inference
• Gradio for hosting the web UI
• Optional: external control nets (Gligen, InstanceDiffusion)
Relationships and Data Flow
1. User → CLI/Gradio UI (text prompt)
2. UI → Pipeline + Memory Manager → construct LLM payload
3. Pipeline → LLM Model → returns Canvas API code
4. Pipeline → execute/parse Canvas calls → Canvas data structure
6. Region Prompter → manipulate attention masks/scores per region
7. Diffusion Model → generate image latent → decode → final image
8. Image → UI → User
Architectural Patterns & Principles
• Pipeline Pattern: clear sequencing of Text → Code → DataModel → Render
• Separation of Concerns: UI vs. LLM orchestration vs. Canvas logic vs. renderer
• Plugin-like LLM Backend: interchangeable model providers (Hugging Face, local)
• Data-Driven Composition: Canvas calls as schema-driven DSL for image composition