How nxp/optimum-ara works
Optimum Ara adapts Transformers-style model APIs to compiled Ara DVM inference: an application supplies inputs to a registered model wrapper, which prepares and generates text or multimodal output through the Ara runtime. Major branches include causal language models, speech seq2seq, and vision-language models with image/video processing. The README documents broader supported-model coverage; sampled source confirms core inference and several generation paths, but not every registration or subsystem connection.
Main entry points: modeling_decoder.py Causal generation · core.py Session and inference · modeling_visual_language.py Visual-language modeling · modeling_seq2seq.py Seq2seq modeling
Components 23
API and configuration
- Package registration
optimum/ara/__init__.py - Ara configuration
optimum/ara/configuration_utils.py - Core configuration
optimum/ara/api/core_configurations.py
Model integrations
- Causal generation
optimum/ara/modeling_decoder.py - Seq2seq modeling
optimum/ara/modeling_seq2seq.py - Visual-language modeling
optimum/ara/modeling_visual_language.py - Llama integration
optimum/ara/models/llama/modeling_llama.py - Qwen integration
optimum/ara/models/qwen/modeling_qwen.py - Whisper integration
optimum/ara/models/whisper/modeling_whisper.py - LLaVA integration
optimum/ara/models/llava/modeling_llava.py - Qwen2.5-VL integration
optimum/ara/models/qwen2_5_vl/modeling_qwen2_5_vl.py - Qwen3-VL integration
optimum/ara/models/qwen3_vl/modeling_qwen3_vl.py
Generation and processing
- Generation configuration
optimum/ara/generation/configuration_utils.py - Logits processing
optimum/ara/generation/logits_process.py - Custom logits processors
optimum/ara/generation/custom_logits_processor.py - Image/video processing
optimum/ara/models/qwen2_5_vl/processing_qwen2_5_vl.py
Ara inference runtime
- Session and inference
optimum/ara/api/core.py
Other components
- Application
- Transformers auto-classes
- Compiled DVM model
- Ara runtime and proxy
- Ara DNPU
- Generated output
Connections 31
- Application → Transformers auto-classes: uses API
- Package registration → Transformers auto-classes: registers models
- Transformers auto-classes → Llama integration: dispatches
- Transformers auto-classes → Qwen integration: dispatches
- Application → Whisper integration: requests inference
- Application → LLaVA integration: requests inference
- Application → Qwen2.5-VL integration: requests inference
- Application → Qwen3-VL integration: requests inference
- Ara configuration → Core configuration: configures
- Causal generation → Generation configuration: prepares generation
- Causal generation → Logits processing: applies processors
- Causal generation → Session and inference: forwards inference
- Seq2seq modeling → Session and inference: forwards inference
- Visual-language modeling → Session and inference: forwards inference
- Llama integration → Causal generation: extends
- Qwen integration → Causal generation: extends
- Whisper integration → Seq2seq modeling: extends
- LLaVA integration → Visual-language modeling: extends
- Qwen2.5-VL integration → Visual-language modeling: extends
- Qwen3-VL integration → Visual-language modeling: extends
- Qwen2.5-VL integration → Image/video processing: prepares inputs
- Qwen3-VL integration → Image/video processing: uses processing
- Causal generation → Custom logits processors: uses processors
- Session and inference → Core configuration: reads settings
- Session and inference → Ara runtime and proxy: creates session
- Session and inference → Compiled DVM model: loads model
- Session and inference → Ara runtime and proxy: sends inference
- Ara runtime and proxy → Ara DNPU: dispatches inference
- Causal generation → Generated output: returns tokens
- Seq2seq modeling → Generated output: returns output
- Visual-language modeling → Generated output: returns output