How espressif/esp-gmf works
1. Repository shape
- This is a multi-component Espressif multimedia monorepo built around the ESP-IDF component model.
- The top-level split is: gmf_core for the runtime framework, elements for reusable processing blocks, packages for higher-level solutions, gmf_examples for reference apps, plus tools and CI/test apps.
2. Core framework boundary: gmf_core
- gmf_core is the foundation layer that everything else builds on.
- It provides the pipeline runtime, element abstraction, ports, payloads, events, tasks, queues, pools, and data buses/ring buffers/FIFOs.
- It also includes the object/capability/method system used for negotiation and dynamic configuration between components.
3. Runtime and portability layer
- The core has an OAL layer (oal/) that wraps memory, mutex, thread, and system primitives.
- This keeps the framework portable across ESP targets while the higher layers remain focused on media flow rather than RTOS details.
- The helpers/ area provides URI parsing and capability/FourCC utilities used during stream setup.
4. Atomic processing elements: elements/
- The elements directory contains the low-level building blocks that plug into gmf_core.
- gmf_audio covers audio decode/encode, resampling, channel conversion, effects, mixing, and time-domain transforms.
- gmf_video covers video decode/encode and image operations like scale, crop, rotate, FPS conversion, overlay, and color conversion.
- gmf_io provides file, HTTP, embedded flash, and I2S/PDM access.
- gmf_ai_audio adds speech-oriented processing such as AEC and wake-word related functions.
- gmf_misc provides small utility elements like copier blocks.
5. How data moves through the framework
- The basic flow is: IO source → pipeline element(s) → processing/effect elements → sink or device IO.
- Negotiation happens before steady-state flow so connected elements agree on caps such as sample rate, format, channels, or video geometry.
- The core’s ports, queues, and payload containers are the main in-memory handoff boundaries between elements.
6. Higher-level application packages: packages/
- Packages compose multiple elements into ready-to-use subsystems.
- They hide pipeline assembly, board/peripheral setup, and common configuration patterns from application code.
- This layer is where most end-user functionality lives: playback, capture, rendering, Bluetooth audio, FFT utilities, and board auto-configuration.
7. Audio-centric packages
- esp_audio_simple_player is a simplified playback stack built on gmf_audio + gmf_io.
- esp_audio_render builds a rendering/mixing pipeline for output-oriented audio use cases.
- esp_bt_audio integrates Bluetooth classic audio profiles with stream APIs, and can optionally bridge into GMF IO.
- These packages sit above the core and elements layers, turning reusable blocks into application-ready flows.
8. Capture and render subsystems
- esp_capture is a high-level capture framework that abstracts audio/video source selection, capture paths, synchronization, and muxing.
- It has specialized implementations for device sources, file sources, GMF-based paths, text overlay, and path managers.
- esp_video_render is the video/UI composition subsystem, with pipelines, dual-stream support, widget layers, backends, and blending/compose logic.
- Its backends target LCD/LVGL-style display integration, while UI primitives live under the vui/ widgets.
9. Board and peripheral abstraction
- esp_board_manager is a major orchestration layer for hardware bring-up.
- It uses YAML board/device/peripheral descriptions plus generators to synthesize board metadata, SDK config, and peripheral setup.
- It abstracts board-specific details such as audio codecs, displays, GPIO expanders, storage, power control, and touch interfaces.
- This layer is what lets examples and packages stay portable across many Espressif boards.
10. Framework composition helpers
- gmf_loader is a configuration/bootstrap package that selects and wires GMF pools/components based on Kconfig.
- gmf_app_utils supplies application scaffolding such as peripheral setup helpers, CLI helpers, system utilities, and unit-test support.
- gmf_fft is a standalone algorithm package for fixed-point FFT/IFFT, separate from the pipeline runtime.
11. Examples and test applications
- Each major component has its own examples/ and/or test_apps/ folder.
- These are end-to-end integration points showing how pipelines, board setup, partition tables, and prebuild scripts fit together on real hardware.
- Most examples follow the same pattern: a component-specific main/ app, idf_component.yml, sdkconfig.defaults, partition CSVs, and Python pytest wrappers for automation.
12. Build, validation, and automation
- The repository is built and tested through ESP-IDF component builds, with CMake, Kconfig, and per-component idf_component.yml manifests.
- CI is split across GitHub Actions and GitLab CI, with separate build/test/deploy workflows and rules files.
- Python tooling under tools/ci and tools/audio_test_package supports automated app builds, board checks, and audio test workflows.
13. Documentation and contribution boundary
- The repo includes component-level README files, changelogs, board docs, and generated diagrams.
- Documentation mirrors the architecture: core, elements, packages, and examples are treated as distinct layers.
- This is useful when drawing the architecture diagram because the repo is intentionally organized around reusable layers rather than a single application.
Main entry points: esp_gmf_element.h Element · esp_gmf_pipeline.h Pipeline · esp_gmf_io_file.c IO Elements · play_http_music.c Examples
Components 24
GMF Core
Runtime framework, object model, ports, pools, queues, and data buses.
- Pipeline · Coordinates connected elements into a flowing media pipeline.
gmf_core/include/esp_gmf_pipeline.h - Element · Common element lifecycle and processing interface.
gmf_core/include/esp_gmf_element.h - Ports · Input/output boundaries used to move payloads between elements.
gmf_core/include/esp_gmf_port.h - Payload · Media data container passed through the framework.
gmf_core/include/esp_gmf_payload.h - Caps & Methods · Capability, method, and object system for setup and dynamic configuration.
gmf_core/include/esp_gmf_cap.h - Data Bus · Queues, ring buffers, FIFOs, and blocks for in-memory transport.
gmf_core/data_bus/esp_gmf_data_bus.c - OAL · Memory, mutex, thread, and system wrappers for ESP portability.
gmf_core/oal/esp_gmf_oal_thread.c - Helpers · URI parsing and FourCC/capability helpers used during stream setup.
gmf_core/helpers/esp_gmf_uri_parser.c
Elements
Reusable atomic processing blocks for audio, video, IO, AI audio, and misc utilities.
- Audio Elements · Decode/encode, rate conversion, channel conversion, mixing, and effects.
elements/gmf_audio/esp_gmf_audio_dec.c - Video Elements · Decode/encode and image operations such as scale, crop, rotate, and convert.
elements/gmf_video/esp_gmf_video_dec.c - IO Elements · File, HTTP, embedded flash, and I2S/PDM access blocks.
elements/gmf_io/esp_gmf_io_file.c - AI Audio · AEC, wake-word, and related speech-oriented audio processing.
elements/gmf_ai_audio/src/esp_gmf_aec.c - Misc Elements · Small helper elements such as copier blocks.
elements/gmf_misc/esp_gmf_copier.c
Packages
Higher-level subsystems that assemble elements into ready-to-use capabilities.
- Simple Player · Simplified audio playback stack built from GMF audio and IO blocks.
packages/esp_audio_simple_player/src/esp_audio_simple_player.c - Audio Render · Audio rendering and mixing pipeline assembly for output-oriented use cases.
packages/esp_audio_render/src/esp_audio_render.c - BT Audio · Bluetooth classic audio profiles with GMF stream integration.
packages/esp_bt_audio/src/esp_bt_audio.c - Capture · Audio/video capture orchestration, source selection, sync, and muxing.
packages/esp_capture/src/esp_capture.c - Video Render · Video composition, dual-stream rendering, backends, and widget layers.
packages/esp_video_render/src/esp_video_render.c - Board Manager · YAML-driven board, device, and peripheral bring-up and generation.
packages/esp_board_manager/src/esp_board_manager.c
Examples & Tests
End-to-end apps, integration tests, and component validation targets.
Tooling
Loader, app utilities, CI scripts, and audio test automation.
- Loader · Kconfig-driven component and pool wiring for GMF defaults.
packages/gmf_loader/src/gmf_loader_setup_all_defaults.c - App Utils · Peripheral setup helpers, CLI support, system utilities, and unit-test helpers.
packages/gmf_app_utils/src/esp_gmf_app_sys.c - FFT · Standalone fixed-point FFT and IFFT implementation.
packages/gmf_fft/src/esp_c/gmf_fft_radix2_dit_s16.c
Connections 31
- Pipeline → Element: runs · Pipelines coordinate element execution and ordering.
- Element → Ports: connects · Elements expose ports for upstream/downstream handoff.
- Ports → Payload: moves · Ports transport payload containers between blocks.
- Element → Caps & Methods: negotiates · Elements exchange capabilities and methods before streaming.
- Data Bus → Payload: buffers · Queues and ring buffers hold payload data in flight.
- OAL → Pipeline: supports · Portable OS wrappers underpin runtime tasks and synchronization.
- Helpers → Caps & Methods: assists · URI and FourCC helpers support capability setup.
- Audio Elements → Element: implements · Audio modules plug into the GMF element model.
- Video Elements → Element: implements · Video modules plug into the GMF element model.
- IO Elements → Element: implements · IO modules plug into the GMF element model.
- AI Audio → Element: implements · AI audio modules plug into the GMF element model.
- Misc Elements → Element: implements · Misc utility blocks plug into the GMF element model.
- Simple Player → Audio Elements: composes · Playback stack is built from audio processing blocks.
- Simple Player → IO Elements: sources · Playback reads media from file, HTTP, or embedded IO.
- Audio Render → Audio Elements: composes · Render package assembles audio transforms into output pipelines.
- BT Audio → IO Elements: bridges · Bluetooth audio can bridge into GMF IO streams.
- Capture → IO Elements: sources · Capture abstracts file, device, and GMF-based sources.
- Capture → Pipeline: builds · Capture path manager assembles GMF pipelines.
- Video Render → Video Elements: composes · Video render uses decode and image-processing elements.
- Video Render → Pipeline: orchestrates · Rendering package coordinates pipelines and worker flow.
- Board Manager → IO Elements: configures · Board manager prepares codec, display, storage, and control devices.
- Board Manager → Examples: enables · Examples depend on board metadata and peripheral setup.
- Loader → Pipeline: bootstraps · Loader selects and wires GMF defaults at startup.
- Loader → Element: registers · Loader exposes component sets used by applications.
- App Utils → Board Manager: supports · App utilities provide setup and CLI helpers for boards.
- Examples → Simple Player: uses · Examples demonstrate ready-made playback stacks.
- Examples → Capture: uses · Examples demonstrate capture and muxing flows.
- Examples → Video Render: uses · Examples demonstrate rendering and widget composition.
- Test Apps → Pipeline: verifies · Test apps exercise core runtime behavior.
- Test Apps → Board Manager: verifies · Integration tests cover board and peripheral setup.
- FFT → App Utils: utility · Standalone algorithm package used independently of the pipeline runtime.