How taoliveaigc/taomate-h3 works
TaoMate-H3 is a distributed, low-latency streaming text-to-audio-video generation runtime. The main workflow accepts prompts and generation settings, launches local GPU workers, performs chunked Stage3 inference with continuity state, then decodes and publishes video. The graph preserves the documented audio-guidance, resolution, and parallel-inference capabilities. Some internal links use README or SOURCE INDEX evidence because the relevant source was not excerpted; uncertain implementation details are not expanded.
Main entry points: pipeline.py H3 pipeline · direct.py Direct runtime · runner.py Run coordinator · runtime.py Chunk generation
Components 31
Request and orchestration
- CLI
src/taomate_h3/cli.py - Run configuration
src/taomate_h3/config.py - Run coordinator
src/taomate_h3/runner.py - GPU worker
src/taomate_h3/worker.py
Streaming generation
- Chunk generation
src/taomate_h3/streaming/runtime.py - Stream session
src/taomate_h3/streaming/session.py - Clean KV cache
src/taomate_h3/streaming/cache.py - Chunk geometry
src/taomate_h3/streaming/geometry.py - Attention hook
src/taomate_h3/streaming/attention_hook.py
Model inference
- Direct runtime
src/taomate_h3/runtime/direct.py - H3 pipeline
src/taomate_h3/model/pipeline.py - Text encoding
src/taomate_h3/model/text_encoder.py - H3 transformer
src/taomate_h3/model/dit.py - Denoising
src/taomate_h3/model/denoise.py - Model architecture
src/taomate_h3/model/architecture.py - Transformer layers
src/taomate_h3/model/layers.py - Packed sequences
src/taomate_h3/model/packed_sequence.py - AV geometry
src/taomate_h3/model/geometry.py - Audio guidance
src/taomate_h3/teacher.py - Teacher denoising
src/taomate_h3/inference/base10_teacher.py - Denoise schedule
src/taomate_h3/denoise_schedule.py - Fused kernels
src/taomate_h3/inference/fused_kernels.py - W8A8 acceleration
src/taomate_h3/inference/w8a8.py - LoRA loading
src/taomate_h3/inference/lora_checkpoint.py
Parallel execution
- TP and Ulysses
src/taomate_h3/distributed/parallel.py - Sharded linear layers
src/taomate_h3/distributed/linear.py - Collectives
src/taomate_h3/distributed/collectives.py - Sequence parallelism
src/taomate_h3/distributed/ulysses.py
Media output
- VAE and output
src/taomate_h3/model/output.py - Published video
Other components
- User
Connections 33
- User → CLI: provides prompts
- CLI → Run configuration: parses settings
src/taomate_h3/cli.py - CLI → Run coordinator: starts run
src/taomate_h3/cli.py - Run coordinator → GPU worker: launches workers
src/taomate_h3/worker.py - Run coordinator → Direct runtime: invokes runtime
src/taomate_h3/runner.py - Direct runtime → H3 pipeline: loads pipeline
src/taomate_h3/runtime/direct.py - Direct runtime → Chunk generation: generates chunks
- Chunk generation → Chunk geometry: plans chunks
src/taomate_h3/streaming/runtime.py - Chunk generation → Stream session: manages session
src/taomate_h3/streaming/runtime.py - Stream session → Attention hook: installs hook
src/taomate_h3/streaming/session.py - Stream session → Clean KV cache: commits KV
src/taomate_h3/streaming/session.py - Chunk generation → Denoising: runs denoising
src/taomate_h3/streaming/runtime.py - H3 pipeline → Text encoding: encodes prompts
src/taomate_h3/model/pipeline.py - H3 pipeline → H3 transformer: runs transformer
src/taomate_h3/model/pipeline.py - H3 pipeline → VAE and output: decodes latents
src/taomate_h3/model/pipeline.py - H3 pipeline → Packed sequences: packs inputs
src/taomate_h3/model/pipeline.py - H3 pipeline → AV geometry: uses geometry
src/taomate_h3/model/pipeline.py - Denoising → Model architecture: uses architecture
src/taomate_h3/model/denoise.py - H3 transformer → Transformer layers: builds layers
src/taomate_h3/model/dit.py - Transformer layers → Attention hook: dispatches attention
src/taomate_h3/model/layers.py - Run coordinator → Audio guidance: prepares guidance
src/taomate_h3/runner.py - Audio guidance → Teacher denoising: uses teacher model
src/taomate_h3/teacher.py - Audio guidance → H3 pipeline: encodes guidance
src/taomate_h3/teacher.py - Audio guidance → Denoise schedule: uses schedule
src/taomate_h3/teacher.py - Direct runtime → TP and Ulysses: sets topology
src/taomate_h3/runtime/direct.py - H3 transformer → Sharded linear layers: uses sharded layers
src/taomate_h3/model/dit.py - Sharded linear layers → Collectives: uses collectives
src/taomate_h3/distributed/linear.py - H3 transformer → Sequence parallelism: uses sequence parallelism
src/taomate_h3/model/dit.py - Transformer layers → Fused kernels: uses fused ops
src/taomate_h3/model/layers.py - Direct runtime → W8A8 acceleration: configures acceleration
src/taomate_h3/runtime/direct.py - Direct runtime → LoRA loading: loads adapter
src/taomate_h3/runtime/direct.py - Run coordinator → Published video: writes video
- VAE and output → Published video: publishes media