# espressif/esp-gmf architecture

> Architecture diagram of the GitHub repository espressif/esp-gmf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/espressif/esp-gmf
- Repository: https://github.com/espressif/esp-gmf

## Overview

1. **Repository shape**
   - This is a multi-component Espressif multimedia monorepo built around the ESP-IDF component model.
   - The top-level split is: **gmf_core** for the runtime framework, **elements** for reusable processing blocks, **packages** for higher-level solutions, **gmf_examples** for reference apps, plus **tools** and **CI/test apps**.

2. **Core framework boundary: `gmf_core`**
   - `gmf_core` is the foundation layer that everything else builds on.
   - It provides the pipeline runtime, element abstraction, ports, payloads, events, tasks, queues, pools, and data buses/ring buffers/FIFOs.
   - It also includes the object/capability/method system used for negotiation and dynamic configuration between components.

3. **Runtime and portability layer**
   - The core has an OAL layer (`oal/`) that wraps memory, mutex, thread, and system primitives.
   - This keeps the framework portable across ESP targets while the higher layers remain focused on media flow rather than RTOS details.
   - The `helpers/` area provides URI parsing and capability/FourCC utilities used during stream setup.

4. **Atomic processing elements: `elements/`**
   - The `elements` directory contains the low-level building blocks that plug into `gmf_core`.
   - `gmf_audio` covers audio decode/encode, resampling, channel conversion, effects, mixing, and time-domain transforms.
   - `gmf_video` covers video decode/encode and image operations like scale, crop, rotate, FPS conversion, overlay, and color conversion.
   - `gmf_io` provides file, HTTP, embedded flash, and I2S/PDM access.
   - `gmf_ai_audio` adds speech-oriented processing such as AEC and wake-word related functions.
   - `gmf_misc` provides small utility elements like copier blocks.

5. **How data moves through the framework**
   - The basic flow is: **IO source → pipeline element(s) → processing/effect elements → sink or device IO**.
   - Negotiation happens before steady-state flow so connected elements agree on caps such as sample rate, format, channels, or video geometry.
   - The core’s ports, queues, and payload containers are the main in-memory handoff boundaries between elements.

6. **Higher-level application packages: `packages/`**
   - Packages compose multiple elements into ready-to-use subsystems.
   - They hide pipeline assembly, board/peripheral setup, and common configuration patterns from application code.
   - This layer is where most end-user functionality lives: playback, capture, rendering, Bluetooth audio, FFT utilities, and board auto-configuration.

7. **Audio-centric packages**
   - `esp_audio_simple_player` is a simplified playback stack built on `gmf_audio` + `gmf_io`.
   - `esp_audio_render` builds a rendering/mixing pipeline for output-oriented audio use cases.
   - `esp_bt_audio` integrates Bluetooth classic audio profiles with stream APIs, and can optionally bridge into GMF IO.
   - These packages sit above the core and elements layers, turning reusable blocks into application-ready flows.

8. **Capture and render subsystems**
   - `esp_capture` is a high-level capture framework that abstracts audio/video source selection, capture paths, synchronization, and muxing.
   - It has specialized implementations for device sources, file sources, GMF-based paths, text overlay, and path managers.
   - `esp_video_render` is the video/UI composition subsystem, with pipelines, dual-stream support, widget layers, backends, and blending/compose logic.
   - Its backends target LCD/LVGL-style display integration, while UI primitives live under the `vui/` widgets.

9. **Board and peripheral abstraction**
   - `esp_board_manager` is a major orchestration layer for hardware bring-up.
   - It uses YAML board/device/peripheral descriptions plus generators to synthesize board metadata, SDK config, and peripheral setup.
   - It abstracts board-specific details such as audio codecs, displays, GPIO expanders, storage, power control, and touch interfaces.
   - This layer is what lets examples and packages stay portable across many Espressif boards.

10. **Framework composition helpers**
   - `gmf_loader` is a configuration/bootstrap package that selects and wires GMF pools/components based on Kconfig.
   - `gmf_app_utils` supplies application scaffolding such as peripheral setup helpers, CLI helpers, system utilities, and unit-test support.
   - `gmf_fft` is a standalone algorithm package for fixed-point FFT/IFFT, separate from the pipeline runtime.

11. **Examples and test applications**
   - Each major component has its own `examples/` and/or `test_apps/` folder.
   - These are end-to-end integration points showing how pipelines, board setup, partition tables, and prebuild scripts fit together on real hardware.
   - Most examples follow the same pattern: a component-specific `main/` app, `idf_component.yml`, `sdkconfig.defaults`, partition CSVs, and Python pytest wrappers for automation.

12. **Build, validation, and automation**
   - The repository is built and tested through ESP-IDF component builds, with CMake, Kconfig, and per-component `idf_component.yml` manifests.
   - CI is split across GitHub Actions and GitLab CI, with separate build/test/deploy workflows and rules files.
   - Python tooling under `tools/ci` and `tools/audio_test_package` supports automated app builds, board checks, and audio test workflows.

13. **Documentation and contribution boundary**
   - The repo includes component-level README files, changelogs, board docs, and generated diagrams.
   - Documentation mirrors the architecture: core, elements, packages, and examples are treated as distinct layers.
   - This is useful when drawing the architecture diagram because the repo is intentionally organized around reusable layers rather than a single application.

Main entry points: [`gmf_core/include/esp_gmf_element.h`](https://github.com/espressif/esp-gmf/blob/main/gmf_core/include/esp_gmf_element.h) (Element), [`gmf_core/include/esp_gmf_pipeline.h`](https://github.com/espressif/esp-gmf/blob/main/gmf_core/include/esp_gmf_pipeline.h) (Pipeline), [`elements/gmf_io/esp_gmf_io_file.c`](https://github.com/espressif/esp-gmf/blob/main/elements/gmf_io/esp_gmf_io_file.c) (IO Elements), [`gmf_examples/basic_examples/pipeline_play_http_music/main/play_http_music.c`](https://github.com/espressif/esp-gmf/blob/main/gmf_examples/basic_examples/pipeline_play_http_music/main/play_http_music.c) (Examples)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["GMF Core"]
  node_gmf_pipeline["Pipeline<br/>runtime graph<br/>[esp_gmf_pipeline.h]"]
  node_gmf_element["Element<br/>base abstraction<br/>[esp_gmf_element.h]"]
  node_gmf_port["Ports<br/>handoff interface<br/>[esp_gmf_port.h]"]
  node_gmf_payload["Payload<br/>data container<br/>[esp_gmf_payload.h]"]
  node_gmf_negotiate["Caps & Methods<br/>negotiation model<br/>[esp_gmf_cap.h]"]
  node_gmf_data_bus["Data Bus<br/>buffering layer<br/>[esp_gmf_data_bus.c]"]
  node_gmf_oal["OAL<br/>portability layer"]
  node_gmf_helpers["Helpers<br/>setup utilities"]
end

subgraph group_elements["Elements"]
  node_audio_elements["Audio Elements<br/>audio transforms"]
  node_video_elements["Video Elements<br/>video transforms"]
  node_io_elements["IO Elements<br/>stream sources<br/>[esp_gmf_io_file.c]"]
  node_ai_audio_elements["AI Audio<br/>speech processing<br/>[esp_gmf_aec.c]"]
  node_misc_elements["Misc Elements<br/>utility blocks<br/>[esp_gmf_copier.c]"]
end

subgraph group_packages["Packages"]
  node_audio_player["Simple Player<br/>playback package"]
  node_audio_render["Audio Render<br/>render package<br/>[esp_audio_render.c]"]
  node_bt_audio["BT Audio<br/>bluetooth package<br/>[esp_bt_audio.c]"]
  node_capture["Capture<br/>[esp_capture.c]"]
  node_video_render["Video Render<br/>ui/render package<br/>[esp_video_render.c]"]
  node_board_mgr["Board Manager<br/>board orchestration"]
end

subgraph group_apps["Examples & Tests"]
  node_examples["Examples<br/>integration apps<br/>[play_http_music.c]"]
  node_test_apps["Test Apps<br/>validation apps"]
end

subgraph group_tooling["Tooling"]
  node_loader["Loader<br/>config bootstrap"]
  node_app_utils["App Utils<br/>app scaffolding<br/>[esp_gmf_app_sys.c]"]
  node_fft["FFT<br/>algorithm package"]
end

node_gmf_pipeline -->|"runs"| node_gmf_element
node_gmf_element -->|"connects"| node_gmf_port
node_gmf_port -->|"moves"| node_gmf_payload
node_gmf_element -->|"negotiates"| node_gmf_negotiate
node_gmf_data_bus -->|"buffers"| node_gmf_payload
node_gmf_oal -.->|"supports"| node_gmf_pipeline
node_gmf_helpers -.->|"assists"| node_gmf_negotiate
node_audio_elements -->|"implements"| node_gmf_element
node_video_elements -->|"implements"| node_gmf_element
node_io_elements -->|"implements"| node_gmf_element
node_ai_audio_elements -->|"implements"| node_gmf_element
node_misc_elements -->|"implements"| node_gmf_element
node_audio_player -->|"composes"| node_audio_elements
node_audio_player -->|"sources"| node_io_elements
node_audio_render -->|"composes"| node_audio_elements
node_bt_audio -->|"bridges"| node_io_elements
node_capture -->|"sources"| node_io_elements
node_capture -->|"builds"| node_gmf_pipeline
node_video_render -->|"composes"| node_video_elements
node_video_render -->|"orchestrates"| node_gmf_pipeline
node_board_mgr -->|"configures"| node_io_elements
node_board_mgr -.->|"enables"| node_examples
node_loader -->|"bootstraps"| node_gmf_pipeline
node_loader -.->|"registers"| node_gmf_element
node_app_utils -.->|"supports"| node_board_mgr
node_examples -->|"uses"| node_audio_player
node_examples -->|"uses"| node_capture
node_examples -->|"uses"| node_video_render
node_test_apps -->|"verifies"| node_gmf_pipeline
node_test_apps -.->|"verifies"| node_board_mgr
node_fft -.->|"utility"| node_app_utils

click node_gmf_pipeline "https://github.com/espressif/esp-gmf/blob/main/gmf_core/include/esp_gmf_pipeline.h"
click node_gmf_element "https://github.com/espressif/esp-gmf/blob/main/gmf_core/include/esp_gmf_element.h"
click node_gmf_port "https://github.com/espressif/esp-gmf/blob/main/gmf_core/include/esp_gmf_port.h"
click node_gmf_payload "https://github.com/espressif/esp-gmf/blob/main/gmf_core/include/esp_gmf_payload.h"
click node_gmf_negotiate "https://github.com/espressif/esp-gmf/blob/main/gmf_core/include/esp_gmf_cap.h"
click node_gmf_data_bus "https://github.com/espressif/esp-gmf/blob/main/gmf_core/data_bus/esp_gmf_data_bus.c"
click node_gmf_oal "https://github.com/espressif/esp-gmf/blob/main/gmf_core/oal/esp_gmf_oal_thread.c"
click node_gmf_helpers "https://github.com/espressif/esp-gmf/blob/main/gmf_core/helpers/esp_gmf_uri_parser.c"
click node_audio_elements "https://github.com/espressif/esp-gmf/blob/main/elements/gmf_audio/esp_gmf_audio_dec.c"
click node_video_elements "https://github.com/espressif/esp-gmf/blob/main/elements/gmf_video/esp_gmf_video_dec.c"
click node_io_elements "https://github.com/espressif/esp-gmf/blob/main/elements/gmf_io/esp_gmf_io_file.c"
click node_ai_audio_elements "https://github.com/espressif/esp-gmf/blob/main/elements/gmf_ai_audio/src/esp_gmf_aec.c"
click node_misc_elements "https://github.com/espressif/esp-gmf/blob/main/elements/gmf_misc/esp_gmf_copier.c"
click node_audio_player "https://github.com/espressif/esp-gmf/blob/main/packages/esp_audio_simple_player/src/esp_audio_simple_player.c"
click node_audio_render "https://github.com/espressif/esp-gmf/blob/main/packages/esp_audio_render/src/esp_audio_render.c"
click node_bt_audio "https://github.com/espressif/esp-gmf/blob/main/packages/esp_bt_audio/src/esp_bt_audio.c"
click node_capture "https://github.com/espressif/esp-gmf/blob/main/packages/esp_capture/src/esp_capture.c"
click node_video_render "https://github.com/espressif/esp-gmf/blob/main/packages/esp_video_render/src/esp_video_render.c"
click node_board_mgr "https://github.com/espressif/esp-gmf/blob/main/packages/esp_board_manager/src/esp_board_manager.c"
click node_loader "https://github.com/espressif/esp-gmf/blob/main/packages/gmf_loader/src/gmf_loader_setup_all_defaults.c"
click node_app_utils "https://github.com/espressif/esp-gmf/blob/main/packages/gmf_app_utils/src/esp_gmf_app_sys.c"
click node_fft "https://github.com/espressif/esp-gmf/blob/main/packages/gmf_fft/src/esp_c/gmf_fft_radix2_dit_s16.c"
click node_examples "https://github.com/espressif/esp-gmf/blob/main/gmf_examples/basic_examples/pipeline_play_http_music/main/play_http_music.c"
click node_test_apps "https://github.com/espressif/esp-gmf/blob/main/gmf_core/test_apps/main/test_gmf_core_main.c"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_gmf_pipeline,node_gmf_element,node_gmf_port,node_gmf_payload,node_gmf_negotiate,node_gmf_data_bus,node_gmf_oal,node_gmf_helpers toneBlue
class node_audio_elements,node_video_elements,node_io_elements,node_ai_audio_elements,node_misc_elements toneAmber
class node_audio_player,node_audio_render,node_bt_audio,node_capture,node_video_render,node_board_mgr toneMint
class node_examples,node_test_apps toneRose
class node_loader,node_app_utils,node_fft toneIndigo
```

## Components

### GMF Core

Runtime framework, object model, ports, pools, queues, and data buses.

- [Pipeline](https://github.com/espressif/esp-gmf/blob/main/gmf_core/include/esp_gmf_pipeline.h): runtime graph · Coordinates connected elements into a flowing media pipeline. · `gmf_core/include/esp_gmf_pipeline.h`
- [Element](https://github.com/espressif/esp-gmf/blob/main/gmf_core/include/esp_gmf_element.h): base abstraction · Common element lifecycle and processing interface. · `gmf_core/include/esp_gmf_element.h`
- [Ports](https://github.com/espressif/esp-gmf/blob/main/gmf_core/include/esp_gmf_port.h): handoff interface · Input/output boundaries used to move payloads between elements. · `gmf_core/include/esp_gmf_port.h`
- [Payload](https://github.com/espressif/esp-gmf/blob/main/gmf_core/include/esp_gmf_payload.h): data container · Media data container passed through the framework. · `gmf_core/include/esp_gmf_payload.h`
- [Caps & Methods](https://github.com/espressif/esp-gmf/blob/main/gmf_core/include/esp_gmf_cap.h): negotiation model · Capability, method, and object system for setup and dynamic configuration. · `gmf_core/include/esp_gmf_cap.h`
- [Data Bus](https://github.com/espressif/esp-gmf/blob/main/gmf_core/data_bus/esp_gmf_data_bus.c): buffering layer · Queues, ring buffers, FIFOs, and blocks for in-memory transport. · `gmf_core/data_bus/esp_gmf_data_bus.c`
- [OAL](https://github.com/espressif/esp-gmf/blob/main/gmf_core/oal/esp_gmf_oal_thread.c): portability layer · Memory, mutex, thread, and system wrappers for ESP portability. · `gmf_core/oal/esp_gmf_oal_thread.c`
- [Helpers](https://github.com/espressif/esp-gmf/blob/main/gmf_core/helpers/esp_gmf_uri_parser.c): setup utilities · URI parsing and FourCC/capability helpers used during stream setup. · `gmf_core/helpers/esp_gmf_uri_parser.c`

### Elements

Reusable atomic processing blocks for audio, video, IO, AI audio, and misc utilities.

- [Audio Elements](https://github.com/espressif/esp-gmf/blob/main/elements/gmf_audio/esp_gmf_audio_dec.c): audio transforms · Decode/encode, rate conversion, channel conversion, mixing, and effects. · `elements/gmf_audio/esp_gmf_audio_dec.c`
- [Video Elements](https://github.com/espressif/esp-gmf/blob/main/elements/gmf_video/esp_gmf_video_dec.c): video transforms · Decode/encode and image operations such as scale, crop, rotate, and convert. · `elements/gmf_video/esp_gmf_video_dec.c`
- [IO Elements](https://github.com/espressif/esp-gmf/blob/main/elements/gmf_io/esp_gmf_io_file.c): stream sources · File, HTTP, embedded flash, and I2S/PDM access blocks. · `elements/gmf_io/esp_gmf_io_file.c`
- [AI Audio](https://github.com/espressif/esp-gmf/blob/main/elements/gmf_ai_audio/src/esp_gmf_aec.c): speech processing · AEC, wake-word, and related speech-oriented audio processing. · `elements/gmf_ai_audio/src/esp_gmf_aec.c`
- [Misc Elements](https://github.com/espressif/esp-gmf/blob/main/elements/gmf_misc/esp_gmf_copier.c): utility blocks · Small helper elements such as copier blocks. · `elements/gmf_misc/esp_gmf_copier.c`

### Packages

Higher-level subsystems that assemble elements into ready-to-use capabilities.

- [Simple Player](https://github.com/espressif/esp-gmf/blob/main/packages/esp_audio_simple_player/src/esp_audio_simple_player.c): playback package · Simplified audio playback stack built from GMF audio and IO blocks. · `packages/esp_audio_simple_player/src/esp_audio_simple_player.c`
- [Audio Render](https://github.com/espressif/esp-gmf/blob/main/packages/esp_audio_render/src/esp_audio_render.c): render package · Audio rendering and mixing pipeline assembly for output-oriented use cases. · `packages/esp_audio_render/src/esp_audio_render.c`
- [BT Audio](https://github.com/espressif/esp-gmf/blob/main/packages/esp_bt_audio/src/esp_bt_audio.c): bluetooth package · Bluetooth classic audio profiles with GMF stream integration. · `packages/esp_bt_audio/src/esp_bt_audio.c`
- [Capture](https://github.com/espressif/esp-gmf/blob/main/packages/esp_capture/src/esp_capture.c): capture framework · Audio/video capture orchestration, source selection, sync, and muxing. · `packages/esp_capture/src/esp_capture.c`
- [Video Render](https://github.com/espressif/esp-gmf/blob/main/packages/esp_video_render/src/esp_video_render.c): ui/render package · Video composition, dual-stream rendering, backends, and widget layers. · `packages/esp_video_render/src/esp_video_render.c`
- [Board Manager](https://github.com/espressif/esp-gmf/blob/main/packages/esp_board_manager/src/esp_board_manager.c): board orchestration · YAML-driven board, device, and peripheral bring-up and generation. · `packages/esp_board_manager/src/esp_board_manager.c`

### Examples & Tests

End-to-end apps, integration tests, and component validation targets.

- [Examples](https://github.com/espressif/esp-gmf/blob/main/gmf_examples/basic_examples/pipeline_play_http_music/main/play_http_music.c): integration apps · Reference applications demonstrating end-to-end pipelines and board usage. · `gmf_examples/basic_examples/pipeline_play_http_music/main/play_http_music.c`
- [Test Apps](https://github.com/espressif/esp-gmf/blob/main/gmf_core/test_apps/main/test_gmf_core_main.c): validation apps · Component-level integration and unit-test harnesses. · `gmf_core/test_apps/main/test_gmf_core_main.c`

### Tooling

Loader, app utilities, CI scripts, and audio test automation.

- [Loader](https://github.com/espressif/esp-gmf/blob/main/packages/gmf_loader/src/gmf_loader_setup_all_defaults.c): config bootstrap · Kconfig-driven component and pool wiring for GMF defaults. · `packages/gmf_loader/src/gmf_loader_setup_all_defaults.c`
- [App Utils](https://github.com/espressif/esp-gmf/blob/main/packages/gmf_app_utils/src/esp_gmf_app_sys.c): app scaffolding · Peripheral setup helpers, CLI support, system utilities, and unit-test helpers. · `packages/gmf_app_utils/src/esp_gmf_app_sys.c`
- [FFT](https://github.com/espressif/esp-gmf/blob/main/packages/gmf_fft/src/esp_c/gmf_fft_radix2_dit_s16.c): algorithm package · Standalone fixed-point FFT and IFFT implementation. · `packages/gmf_fft/src/esp_c/gmf_fft_radix2_dit_s16.c`

## Connections

- Pipeline → Element: runs · Pipelines coordinate element execution and ordering.
- Element → Ports: connects · Elements expose ports for upstream/downstream handoff.
- Ports → Payload: moves · Ports transport payload containers between blocks.
- Element → Caps & Methods: negotiates · Elements exchange capabilities and methods before streaming.
- Data Bus → Payload: buffers · Queues and ring buffers hold payload data in flight.
- OAL → Pipeline: supports · Portable OS wrappers underpin runtime tasks and synchronization.
- Helpers → Caps & Methods: assists · URI and FourCC helpers support capability setup.
- Audio Elements → Element: implements · Audio modules plug into the GMF element model.
- Video Elements → Element: implements · Video modules plug into the GMF element model.
- IO Elements → Element: implements · IO modules plug into the GMF element model.
- AI Audio → Element: implements · AI audio modules plug into the GMF element model.
- Misc Elements → Element: implements · Misc utility blocks plug into the GMF element model.
- Simple Player → Audio Elements: composes · Playback stack is built from audio processing blocks.
- Simple Player → IO Elements: sources · Playback reads media from file, HTTP, or embedded IO.
- Audio Render → Audio Elements: composes · Render package assembles audio transforms into output pipelines.
- BT Audio → IO Elements: bridges · Bluetooth audio can bridge into GMF IO streams.
- Capture → IO Elements: sources · Capture abstracts file, device, and GMF-based sources.
- Capture → Pipeline: builds · Capture path manager assembles GMF pipelines.
- Video Render → Video Elements: composes · Video render uses decode and image-processing elements.
- Video Render → Pipeline: orchestrates · Rendering package coordinates pipelines and worker flow.
- Board Manager → IO Elements: configures · Board manager prepares codec, display, storage, and control devices.
- Board Manager → Examples: enables · Examples depend on board metadata and peripheral setup.
- Loader → Pipeline: bootstraps · Loader selects and wires GMF defaults at startup.
- Loader → Element: registers · Loader exposes component sets used by applications.
- App Utils → Board Manager: supports · App utilities provide setup and CLI helpers for boards.
- Examples → Simple Player: uses · Examples demonstrate ready-made playback stacks.
- Examples → Capture: uses · Examples demonstrate capture and muxing flows.
- Examples → Video Render: uses · Examples demonstrate rendering and widget composition.
- Test Apps → Pipeline: verifies · Test apps exercise core runtime behavior.
- Test Apps → Board Manager: verifies · Integration tests cover board and peripheral setup.
- FFT → App Utils: utility · Standalone algorithm package used independently of the pipeline runtime.