# lllyasviel/framepack architecture

> Architecture diagram of the GitHub repository lllyasviel/framepack, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lllyasviel/framepack
- Repository: https://github.com/lllyasviel/framepack

## Overview

##### Project type and purpose
- FramePack is a desktop video‑generation application written in Python. It provides a Gradio GUI (demo_gradio.py) that drives a next‐frame prediction model for progressive video synthesis. It is neither a web service nor a microservice suite, but a standalone client with a UI layer, a backend orchestration layer, GPU‐based model inference, and external model downloads from Hugging Face.

##### File structure
- Top level:
- demo_gradio.py: entry point launching the Gradio UI and wiring user controls to the backend.
- requirements.txt: Python dependencies (torch, gradio, diffusers, etc.).
- diffusers_helper/: core library providing:
– bucket_tools.py, memory.py, thread_utils.py, utils.py: GPU memory and threading abstractions.
– hf_login.py: handles Hugging Face authentication and model downloads.
– clip_vision.py, hunyuan.py, k_diffusion/*, pipelines/*: wrappers for different diffusion backends and pipelines.
– gradio/progress_bar.py: custom progress‐bar integration.
– models/: model‐specific code (e.g., hunyuan_video_packed).
– dit_common.py: shared data structures or constants.
- No separate “frontend” or “backend” folders, but demo_gradio.py + gradio submodule form the UI layer; everything else is the backend library.

##### README for additional insights
- Purpose: next‑frame video prediction with constant‑length context packing.
- Tech stack: PyTorch, Gradio, Hugging Face diffusion models (Diffusers, k‑diffusion, Hunyuan), optional attention kernels (xformers, flash‑attn, sage‑attention).
- Runtime: local GPU inference, model downloads from Hugging Face Model Hub.
- GUI: user uploads image & prompt → initiates generation → sees per‐section progress & latent preview.

##### Diagramming recommendations
- Related modules (all diffusers_helper submodules) into a “Backend Library” container.
- UI → Controller → Backend Library
- Backend Library → GPU Memory → GPU Compute
- Backend Library → Hugging Face Model Hub
- Backend Library → Local Cache / Disk
- Backend Library → UI (for progress & preview streaming)
- Frontend (Gradio) – e.g., blue
- Backend library – e.g., green
- External services – e.g., orange
- Infrastructure (GPU, disk) – e.g., gray

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        UI["User / Client UI\n(Gradio Web UI)"]:::frontend
        Controller["UI Controller\n(Glue code)"]:::frontend
    end

    %% Backend Library
    subgraph "Backend Library"
        Utilities["Shared Utils & Entry\n(utils.py, dit_common.py)"]:::backend
        PipelineMgr["Pipeline Manager\n(k_diffusion_hunyuan.py)"]:::backend
        HFDiff["HF Diffusers Wrappers"]:::backend
        KDiff["k‑diffusion Implementation\n(uni_pc_fm.py)"]:::backend
        Hunyuan["Hunyuan Wrapper\n(hunyuan.py)"]:::backend
        CLIP["CLIP Vision Conditioning\n(clip_vision.py)"]:::backend
        MemMgr["Memory Manager\n(memory.py)"]:::backend
        ThreadUtil["Threading Utilities\n(thread_utils.py)"]:::backend
        ProgressUpd["Progress‑Bar Updater\n(progress_bar.py)"]:::backend
        HFLogin["HF Login & Model Download\n(hf_login.py)"]:::backend
        Bucket["Bucket Tools & Cache\n(bucket_tools.py)"]:::backend
        ModelSpec["Model‑specific Code\n(hunyuan_video_packed.py)"]:::backend
    end

    %% External Services
    subgraph "External Services"
        HFHub["Hugging Face Model Hub"]:::external
    end

    %% Infrastructure
    subgraph "Infrastructure"
        GPU["GPU Compute & Memory"]:::infra
        Disk["Local Storage / Cache"]:::infra
    end

    %% Connections
    UI -->|"user actions"| Controller
    Controller -->|"invoke pipeline"| PipelineMgr
    PipelineMgr -->|"model weights request"| HFLogin
    HFLogin -->|"download models"| HFHub
    HFLogin -->|"store weights"| Bucket
    PipelineMgr -->|"select backend"| HFDiff
    PipelineMgr -->|"select backend"| KDiff
    PipelineMgr -->|"select backend"| Hunyuan
    PipelineMgr -->|"conditioning"| CLIP
    PipelineMgr -->|"load to GPU"| MemMgr
    MemMgr --> GPU
    PipelineMgr -->|"spawn tasks"| ThreadUtil
    ThreadUtil --> GPU
    GPU -->|"frame tensor"| ProgressUpd
    ProgressUpd -->|"progress update & previews"| UI
    PipelineMgr -->|"write final frames"| Disk
    Disk -->|"serve frames"| UI

    %% Click Events
    click UI "https://github.com/lllyasviel/framepack/blob/main/demo_gradio.py"
    click Controller "https://github.com/lllyasviel/framepack/blob/main/demo_gradio.py"
    click ProgressUpd "https://github.com/lllyasviel/framepack/blob/main/diffusers_helper/gradio/progress_bar.py"
    click Utilities "https://github.com/lllyasviel/framepack/blob/main/diffusers_helper/utils.py"
    click PipelineMgr "https://github.com/lllyasviel/framepack/blob/main/diffusers_helper/pipelines/k_diffusion_hunyuan.py"
    click HFDiff "https://github.com/lllyasviel/framepack/tree/main/diffusers_helper/pipelines/"
    click KDiff "https://github.com/lllyasviel/framepack/blob/main/diffusers_helper/k_diffusion/uni_pc_fm.py"
    click Hunyuan "https://github.com/lllyasviel/framepack/blob/main/diffusers_helper/hunyuan.py"
    click CLIP "https://github.com/lllyasviel/framepack/blob/main/diffusers_helper/clip_vision.py"
    click MemMgr "https://github.com/lllyasviel/framepack/blob/main/diffusers_helper/memory.py"
    click ThreadUtil "https://github.com/lllyasviel/framepack/blob/main/diffusers_helper/thread_utils.py"
    click HFLogin "https://github.com/lllyasviel/framepack/blob/main/diffusers_helper/hf_login.py"
    click Bucket "https://github.com/lllyasviel/framepack/blob/main/diffusers_helper/bucket_tools.py"
    click ModelSpec "https://github.com/lllyasviel/framepack/blob/main/diffusers_helper/models/hunyuan_video_packed.py"

    %% Styles
    classDef frontend fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef backend fill:#90EE90,stroke:#333,stroke-width:1px
    classDef external fill:#FFA07A,stroke:#333,stroke-width:1px
    classDef infra fill:#D3D3D3,stroke:#333,stroke-width:1px
```
