minimax-ai/minimax-h3

Purpose and Entry Points

MiniMax H3 is a multimodal generative-model distribution for video plus native stereo audio. It ships two locally deployable 768p H3-Base checkpoint families—first/last-frame (FL2VA/) and multimodal-reference (Ref2VA/)—plus API-driven Context-IR and 2K regeneration workflows. Primary repository entry metadata is model_index.json; task-specific model manifests are FL2VA/model_index.json and Ref2VA/model_index.json.

Deployment Boundary

This is a model/checkpoint repository, not an application service implementation. Inference is supplied by external runtimes: SGLang, vLLM, Diffusers, or ComfyUI. The documented local serving entry is SGLang with a task variant, while reproducible request clients are shell scripts such as scripts/readme/reproducible-768p-t2va-request.sh and scripts/readme/reproducible-768p-ref2va-request.sh.

Checkpoint Families

FL2VA/ supports text-to-audio-video and optional first/last-frame conditioning. Ref2VA/ supports text plus images, video, and audio references. Both are self-contained Hugging Face-style component bundles, with duplicated architectural implementations and distinct transformer weights:

• FL2VA/transformer/config.json

• Ref2VA/transformer/config.json

• FL2VA/processor/preprocessor_config.json

Multimodal Input Processing

A processor/tokenizer and pretrained text encoder convert prompt and structured multimodal context into text/vision representations, including special dialogue tokens required by the model. Primary artifacts:

• FL2VA/processor/chat_template.json

• FL2VA/tokenizer/tokenizer_config.json

• FL2VA/text_encoder/config.json

For Ref2VA, input references are bounded externally by the API/workflow contract; modality interpretation and context refinement belong to hosted Context-IR rather than this repository.

Latent Encoding and Decoding

Visual and audio data cross explicit VAE boundaries before and after generation. The visual VAE is temporally causal and contains convolutional, attention, normalization, and ViT/CNN decoder components:

• FL2VA/video_vae/minimax_h3_video_vae.py

• FL2VA/video_vae/klvae.py

• FL2VA/video_vae/vae_vit.py

Audio is independently encoded/decoded per stereo channel through the DAC-based AudioVAE:

• FL2VA/audio_vae/minimax_h3_audio_vae.py

• FL2VA/audio_vae/dac_audio_vae.py

• FL2VA/audio_vae/dac_bigvgan.py

Joint Generation Core

Encoded text, visual, and audio context is packed into one multimodal sequence for the H3 Omni Transformer. The transformer jointly predicts video and audio latents; modality-specific behavior is concentrated at I/O and conditioning branches rather than separate transformer trunks. Weight shard indexes identify externally stored model tensors:

• FL2VA/transformer/model.safetensors.index.json

• Ref2VA/transformer/model.safetensors.index.json

• transformer/diffusion_pytorch_model.safetensors.index.json

Schedulers define the diffusion-generation control schedule:

• scheduler/scheduler_config.json

• audio_scheduler/scheduler_config.json

Full 2K Workflow and External Services

The 2K path is orchestrated across boundaries: hosted H3-Context-IR transforms raw multimodal intent into generation-ready context; local H3-Base produces 768p video/audio; hosted H3-Regenerate-2K consumes the base result plus original context. Context-IR and regeneration are not included locally and require MiniMax Open Platform APIs. Request examples encode this handoff:

• scripts/readme/full-2k-t2va-h3-context-ir.sh

• scripts/readme/full-2k-t2va-h3-base.sh

• scripts/readme/full-2k-t2va-h3-regenerate-2k.sh

Prompt-Authoring Layer

The skills/ hierarchy is an independent prompt-authoring package for users or agents, not inference code. It standardizes structured H3 prompting and style-specific planning before requests reach Context-IR or H3-Base:

• skills/h3-prompt-writing/SKILL.md

• skills/h3-prompt-writing/references/base-en.txt

• skills/h3-prompt-writing/references/ref-en.txt