# aiinpocket/framepack architecture

> Architecture diagram of the GitHub repository aiinpocket/framepack, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aiinpocket/framepack
- Repository: https://github.com/aiinpocket/framepack

## Overview

##### Project Type and Purpose
- FramePack is a standalone desktop application for next-frame prediction video generation. It bundles a Gradio‐based GUI (frontend) with a Python application core that orchestrates video diffusion pipelines, model loading, memory management, and GPU kernel integrations. There's no separate web service—everything runs locally.

##### File Structure Analysis
- Top level
- demo_gradio.py, demo_gradio_f1.py: Entry points for launching the Gradio GUI frontend.
- requirements.txt: Lists Python and ML dependencies (PyTorch, diffusers, gradio, etc.).
- diffusers_helper/ (application core library)
- bucket_tools.py: File I/O and data‐upload utilities (caching, model downloads).
- hf_login.py: HuggingFace authentication and model retrieval.
- memory.py: Memory management strategies (TeaCache, quantization toggles).
- models/: Definitions and wrappers for packed model checkpoints (e.g., hunyuan_video_packed.py).
- pipelines/: Next‐frame‐section prediction pipeline implementations (k_diffusion_hunyuan.py).
- clip_vision.py, utils.py, dit_common.py: Pre‐ and post‐processing, utility routines.
- k_diffusion/: Third‐party diffusion kernels integration via wrapper.py and uni_pc_fm.
- thread_utils.py: Asynchronous task execution (e.g., progressive section generation).
- gradio/progress_bar.py: Custom Gradio components to show in‐UI progress.

##### README Insights
- Describes the overall architecture: GUI front end (Gradio) → next‐frame-section diffusion engine → GPU kernels (PyTorch, xformers, flash‐attn, sage) → progressive frame output with live preview.
- Emphasizes components like TeaCache, various attention kernels, and HuggingFace model hub integration.

a. Main Components
- Presentation Layer: Gradio UI (demo_gradio.py)
- Controller Layer: CLI argument parsing, user settings, event loop
- Core Engine: diffusers_helper package
– Model Loader (hf_login + bucket_tools)
– Pipeline Manager (pipelines/k_diffusion_hunyuan)
– GPU Kernel Interface (k_diffusion.wrapper, PyTorch, xformers)
– Memory Manager (memory.py)
– Utility Services (clip_vision, utils, dit_common)
– Threading & Async (thread_utils)
- Data Layer:
– Local Cache (downloaded model files, buckets)
– Temporary Frame Storage
- External Services:
– HuggingFace Model Hub
– GPU Hardware & Drivers
b. Relationships & Interactions
1. User uploads image and prompt via Gradio →
2. Controller translates UI inputs into pipeline config →
3. Model Loader checks local cache or authenticates via hf_login →
4. Pipeline Manager invokes k_diffusion or other kernels →
5. Memory Manager monitors GPU usage and invokes TeaCache/quantization →
6. Thread Utils dispatches section‐by‐section asynchronous generation →
7. Progress Bar component streams preview frames back to UI →
8. Final video frames are written to disk or displayed in UI.
c. Architectural Patterns & Principles
- Layered Architecture: Clear separation of UI, orchestration, core business logic, infrastructure.
- Modular Design: diffusers_helper as a library with well‐scoped modules (models, pipelines, utils).
- Asynchronous Processing: thread_utils for non‐blocking UI updates and long‐running GPU tasks.
- Strategy Pattern for Kernel Selection: Abstract GPU kernel interface supporting PyTorch, xformers, flash‐attn, sage.
- Resource Management: memory.py handles dynamic adjustment (TeaCache, quantization) to fit large models on limited GPU.
d. Key Technologies
- Python 3.10, PyTorch, HuggingFace Diffusers, Gradio
- k‐diffusion, xformers, flash‐attn, sage‐attention kernels
- HuggingFace Hub for model storage
- Threading for async pipelines

- Blue for presentation (Gradio)
- Green for core logic (diffusers_helper modules)
- Orange for infrastructure (file I/O, GPU kernels)
- Gray for external services

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Presentation Layer
    subgraph "Presentation Layer"
        direction TB
        UI1["Gradio UI\ndemo_gradio.py"]:::presentation
        UI2["Gradio UI\n demo_gradio_f1.py"]:::presentation
        PB["Progress Bar\nprogress_bar.py"]:::presentation
    end

    %% Controller Layer
    subgraph "Controller Layer"
        direction TB
        CTRL1["Arg Parsing\n(demo_gradio)"]:::presentation
        CTRL2["Arg Parsing\n(demo_gradio_f1)"]:::presentation
    end

    %% Core Engine
    subgraph "Core Engine (diffusers_helper)" 
        direction TB

        subgraph "Model Loader" 
            direction TB
            ML1["hf_login.py"]:::core
            ML2["bucket_tools.py"]:::core
        end

        subgraph "Pipeline Manager"
            direction TB
            PM["k_diffusion_hunyuan.py"]:::core
        end

        subgraph "GPU Kernel Interface"
            direction TB
            GK1["wrapper.py"]:::infra
            GK2["uni_pc_fm.py"]:::infra
        end

        subgraph "Memory Manager"
            direction TB
            MEM["memory.py"]:::core
        end

        subgraph "Utility Services"
            direction TB
            US1["clip_vision.py"]:::core
            US2["utils.py"]:::core
            US3["dit_common.py"]:::core
        end

        subgraph "Threading & Async"
            direction TB
            TH["thread_utils.py"]:::core
        end

        Models["hunyuan_video_packed.py"]:::core
    end

    %% Data Layer
    subgraph "Data Layer"
        direction TB
        CACHE["Model Cache"]:::storage
        TEMP["Temp Frame Storage"]:::storage
    end

    %% External Services
    subgraph "External"
        direction TB
        HF["HuggingFace Hub"]:::external
        GPU["GPU Hardware & Drivers"]:::external
    end

    %% Connections
    UI1 -->|"user input"| CTRL1
    UI2 -->|"user input"| CTRL2
    CTRL1 -->|"load_model()"| ML1
    CTRL1 -->|"load_model()"| ML2
    CTRL2 -->|"load_model()"| ML1
    CTRL2 -->|"load_model()"| ML2

    ML1 -->|"auth/model fetch"| HF
    ML2 -->|"cache I/O"| CACHE

    TH -->|"dispatch"| PM
    MEM -->|"monitor/adjust"| GK1
    MEM -->|"monitor/adjust"| GK2

    PM -->|"generate_section()"| GK1
    PM -->|"generate_section()"| GK2
    GK1 -->|"kernel calls"| GPU
    GK2 -->|"kernel calls"| GPU

    PM -->|"write frames"| TEMP
    TEMP -->|"stream_preview()"| PB
    PB -->|"update UI"| UI1
    PB -->|"update UI"| UI2

    TEMP -->|"final output"| UI1
    TEMP -->|"final output"| UI2

    %% Click Events
    click UI1 "https://github.com/aiinpocket/framepack/blob/main/demo_gradio.py"
    click UI2 "https://github.com/aiinpocket/framepack/blob/main/demo_gradio_f1.py"
    click PB "https://github.com/aiinpocket/framepack/blob/main/diffusers_helper/gradio/progress_bar.py"
    click ML1 "https://github.com/aiinpocket/framepack/blob/main/diffusers_helper/hf_login.py"
    click ML2 "https://github.com/aiinpocket/framepack/blob/main/diffusers_helper/bucket_tools.py"
    click PM "https://github.com/aiinpocket/framepack/blob/main/diffusers_helper/pipelines/k_diffusion_hunyuan.py"
    click GK1 "https://github.com/aiinpocket/framepack/blob/main/diffusers_helper/k_diffusion/wrapper.py"
    click GK2 "https://github.com/aiinpocket/framepack/blob/main/diffusers_helper/k_diffusion/uni_pc_fm.py"
    click MEM "https://github.com/aiinpocket/framepack/blob/main/diffusers_helper/memory.py"
    click US1 "https://github.com/aiinpocket/framepack/blob/main/diffusers_helper/clip_vision.py"
    click US2 "https://github.com/aiinpocket/framepack/blob/main/diffusers_helper/utils.py"
    click US3 "https://github.com/aiinpocket/framepack/blob/main/diffusers_helper/dit_common.py"
    click TH "https://github.com/aiinpocket/framepack/blob/main/diffusers_helper/thread_utils.py"
    click Models "https://github.com/aiinpocket/framepack/blob/main/diffusers_helper/models/hunyuan_video_packed.py"

    %% Styles
    classDef presentation fill:#E0F7FA,stroke:#0288D1,stroke-width:2px;
    classDef core fill:#E8F5E9,stroke:#388E3C,stroke-width:2px;
    classDef infra fill:#FFF3E0,stroke:#F57C00,stroke-width:2px;
    classDef external fill:#ECEFF1,stroke:#546E7A,stroke-width:2px;
    classDef storage fill:#FAFAFA,stroke:#000000,stroke-width:1px;
```
