# nxp/optimum-ara architecture

> Architecture diagram of the GitHub repository nxp/optimum-ara, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nxp/optimum-ara
- Repository: https://github.com/nxp/optimum-ara

## Overview

Optimum Ara adapts Transformers-style model APIs to compiled Ara DVM inference: an application supplies inputs to a registered model wrapper, which prepares and generates text or multimodal output through the Ara runtime. Major branches include causal language models, speech seq2seq, and vision-language models with image/video processing. The README documents broader supported-model coverage; sampled source confirms core inference and several generation paths, but not every registration or subsystem connection.

Main entry points: [`optimum/ara/modeling_decoder.py`](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/modeling_decoder.py) (Causal generation), [`optimum/ara/api/core.py`](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/api/core.py) (Session and inference), [`optimum/ara/modeling_visual_language.py`](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/modeling_visual_language.py) (Visual-language modeling), [`optimum/ara/modeling_seq2seq.py`](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/modeling_seq2seq.py) (Seq2seq modeling)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API and configuration"]
  node_ara_init["Package registration<br/>[__init__.py]"]
  node_config["Ara configuration"]
  node_core_config["Core configuration"]
end

subgraph group_models["Model integrations"]
  node_decoder["Causal generation"]
  node_seq2seq["Seq2seq modeling"]
  node_visual["Visual-language modeling"]
  node_llama["Llama integration<br/>[modeling_llama.py]"]
  node_qwen["Qwen integration<br/>[modeling_qwen.py]"]
  node_whisper["Whisper integration"]
  node_llava["LLaVA integration<br/>[modeling_llava.py]"]
  node_qwen25vl["Qwen2.5-VL integration"]
  node_qwen3vl["Qwen3-VL integration"]
end

subgraph group_generation["Generation and processing"]
  node_generation_config["Generation configuration"]
  node_logits["Logits processing<br/>[logits_process.py]"]
  node_custom_logits["Custom logits processors"]
  node_vl_processing["Image/video processing"]
end

subgraph group_runtime["Ara inference runtime"]
  node_core["Session and inference<br/>[core.py]"]
end

node_application(("Application"))
node_transformers["Transformers auto-classes"]
node_compiled_model["Compiled DVM model"]
node_ara_runtime["Ara runtime and proxy"]
node_dnpu{{"Ara DNPU"}}
node_result["Generated output"]

node_application -->|"uses API"| node_transformers
node_ara_init -->|"registers models"| node_transformers
node_transformers -->|"dispatches"| node_llama
node_transformers -->|"dispatches"| node_qwen
node_application -->|"requests inference"| node_whisper
node_application -->|"requests inference"| node_llava
node_application -->|"requests inference"| node_qwen25vl
node_application -->|"requests inference"| node_qwen3vl
node_config -->|"configures"| node_core_config
node_decoder -->|"prepares generation"| node_generation_config
node_decoder -->|"applies processors"| node_logits
node_decoder -->|"forwards inference"| node_core
node_seq2seq -->|"forwards inference"| node_core
node_visual -->|"forwards inference"| node_core
node_llama -->|"extends"| node_decoder
node_qwen -->|"extends"| node_decoder
node_whisper -->|"extends"| node_seq2seq
node_llava -->|"extends"| node_visual
node_qwen25vl -->|"extends"| node_visual
node_qwen3vl -->|"extends"| node_visual
node_qwen25vl -.->|"prepares inputs"| node_vl_processing
node_qwen3vl -.->|"uses processing"| node_vl_processing
node_decoder -.->|"uses processors"| node_custom_logits
node_core -->|"reads settings"| node_core_config
node_core -->|"creates session"| node_ara_runtime
node_core -->|"loads model"| node_compiled_model
node_core -->|"sends inference"| node_ara_runtime
node_ara_runtime -->|"dispatches inference"| node_dnpu
node_decoder -->|"returns tokens"| node_result
node_seq2seq -->|"returns output"| node_result
node_visual -->|"returns output"| node_result

click node_ara_init "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/__init__.py"
click node_config "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/configuration_utils.py"
click node_core_config "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/api/core_configurations.py"
click node_core "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/api/core.py"
click node_decoder "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/modeling_decoder.py"
click node_seq2seq "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/modeling_seq2seq.py"
click node_visual "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/modeling_visual_language.py"
click node_llama "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/models/llama/modeling_llama.py"
click node_qwen "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/models/qwen/modeling_qwen.py"
click node_whisper "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/models/whisper/modeling_whisper.py"
click node_llava "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/models/llava/modeling_llava.py"
click node_qwen25vl "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/models/qwen2_5_vl/modeling_qwen2_5_vl.py"
click node_qwen3vl "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/models/qwen3_vl/modeling_qwen3_vl.py"
click node_generation_config "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/generation/configuration_utils.py"
click node_logits "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/generation/logits_process.py"
click node_custom_logits "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/generation/custom_logits_processor.py"
click node_vl_processing "https://github.com/nxp/optimum-ara/blob/main/optimum/ara/models/qwen2_5_vl/processing_qwen2_5_vl.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ara_init,node_config,node_core_config toneBlue
class node_decoder,node_seq2seq,node_visual,node_llama,node_qwen,node_whisper,node_llava,node_qwen25vl,node_qwen3vl toneAmber
class node_generation_config,node_logits,node_custom_logits,node_vl_processing toneMint
class node_core toneRose
class node_application,node_transformers,node_compiled_model,node_ara_runtime,node_dnpu,node_result toneIndigo
```

## Components

### API and configuration

- [Package registration](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/__init__.py): `optimum/ara/__init__.py`
- [Ara configuration](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/configuration_utils.py): `optimum/ara/configuration_utils.py`
- [Core configuration](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/api/core_configurations.py): `optimum/ara/api/core_configurations.py`

### Model integrations

- [Causal generation](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/modeling_decoder.py): `optimum/ara/modeling_decoder.py`
- [Seq2seq modeling](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/modeling_seq2seq.py): `optimum/ara/modeling_seq2seq.py`
- [Visual-language modeling](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/modeling_visual_language.py): `optimum/ara/modeling_visual_language.py`
- [Llama integration](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/models/llama/modeling_llama.py): `optimum/ara/models/llama/modeling_llama.py`
- [Qwen integration](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/models/qwen/modeling_qwen.py): `optimum/ara/models/qwen/modeling_qwen.py`
- [Whisper integration](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/models/whisper/modeling_whisper.py): `optimum/ara/models/whisper/modeling_whisper.py`
- [LLaVA integration](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/models/llava/modeling_llava.py): `optimum/ara/models/llava/modeling_llava.py`
- [Qwen2.5-VL integration](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/models/qwen2_5_vl/modeling_qwen2_5_vl.py): `optimum/ara/models/qwen2_5_vl/modeling_qwen2_5_vl.py`
- [Qwen3-VL integration](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/models/qwen3_vl/modeling_qwen3_vl.py): `optimum/ara/models/qwen3_vl/modeling_qwen3_vl.py`

### Generation and processing

- [Generation configuration](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/generation/configuration_utils.py): `optimum/ara/generation/configuration_utils.py`
- [Logits processing](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/generation/logits_process.py): `optimum/ara/generation/logits_process.py`
- [Custom logits processors](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/generation/custom_logits_processor.py): `optimum/ara/generation/custom_logits_processor.py`
- [Image/video processing](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/models/qwen2_5_vl/processing_qwen2_5_vl.py): `optimum/ara/models/qwen2_5_vl/processing_qwen2_5_vl.py`

### Ara inference runtime

- [Session and inference](https://github.com/nxp/optimum-ara/blob/main/optimum/ara/api/core.py): `optimum/ara/api/core.py`

### Other components

- **Application**
- **Transformers auto-classes**
- **Compiled DVM model**
- **Ara runtime and proxy**
- **Ara DNPU**
- **Generated output**

## Connections

- Application → Transformers auto-classes: uses API
- Package registration → Transformers auto-classes: registers models
- Transformers auto-classes → Llama integration: dispatches
- Transformers auto-classes → Qwen integration: dispatches
- Application → Whisper integration: requests inference
- Application → LLaVA integration: requests inference
- Application → Qwen2.5-VL integration: requests inference
- Application → Qwen3-VL integration: requests inference
- Ara configuration → Core configuration: configures
- Causal generation → Generation configuration: prepares generation
- Causal generation → Logits processing: applies processors
- Causal generation → Session and inference: forwards inference
- Seq2seq modeling → Session and inference: forwards inference
- Visual-language modeling → Session and inference: forwards inference
- Llama integration → Causal generation: extends
- Qwen integration → Causal generation: extends
- Whisper integration → Seq2seq modeling: extends
- LLaVA integration → Visual-language modeling: extends
- Qwen2.5-VL integration → Visual-language modeling: extends
- Qwen3-VL integration → Visual-language modeling: extends
- Qwen2.5-VL integration → Image/video processing: prepares inputs
- Qwen3-VL integration → Image/video processing: uses processing
- Causal generation → Custom logits processors: uses processors
- Session and inference → Core configuration: reads settings
- Session and inference → Ara runtime and proxy: creates session
- Session and inference → Compiled DVM model: loads model
- Session and inference → Ara runtime and proxy: sends inference
- Ara runtime and proxy → Ara DNPU: dispatches inference
- Causal generation → Generated output: returns tokens
- Seq2seq modeling → Generated output: returns output
- Visual-language modeling → Generated output: returns output