How m5stack/stackchan works
1. Repository shape
- This repo is a multi-target product stack for StackChan: a Flutter mobile app, ESP32-S3 firmware for the robot, a separate ESP32 remote controller firmware, and a Go backend server.
- The architecture is split by runtime and deployment boundary, not by package alone.
2. Core product boundary: the robot
- The main embedded system lives in firmware/ and runs on the M5Stack CoreS3 / ESP32-S3.
- It owns the physical robot: display, camera, audio, servos, IMU, LEDs, touch, RTC, BLE/Wi‑Fi, OTA, and ESP-NOW.
- Its job is to turn hardware events and network commands into robot behavior and to expose device capabilities upward to the app/server.
3. Firmware layering
- firmware/main/hal/ is the hardware abstraction layer: audio, BLE, ESP-NOW, network, OTA, RTC, servo, IMU, camera/display, NFC-like peripherals, and board-specific glue.
- firmware/main/stackchan/ is the robot domain layer: avatar rendering, motion, animation, modifiers, JSON helpers, and utility math.
- firmware/main/apps/ is a mini app framework with separate user-facing modes such as launcher, setup, app center, AI agent, avatar, dance, and EZ data.
4. Robot behavior and state model
- The firmware’s behavior is organized around modifiers and motion/animation primitives.
- Avatar/emotion concepts are modeled separately from servo motion, which lets the robot express states like speaking, blinking, idle motion, dance, and head pet interactions.
- This is a key diagram boundary: “behavior state” is distinct from “hardware actuation.”
5. Mobile app as the orchestration client
- app/lib/ is a Flutter client that coordinates setup, device binding, dance management, conversation, monitoring, music upload, and settings.
- It talks to the backend over HTTP and to devices over network/socket utilities.
- The app is not just a UI shell; it contains the product’s user flows for onboarding, pairing, content management, and live control.
6. Mobile app native integration
- iOS has native views and bridging under app/ios/Runner/, including AR/3D robot views, motion screens, and a NativeBridge.
- Android has a parallel native layer under app/android/ with Kotlin models and a 3D model asset.
- These native layers exist to render the robot locally, not merely to host Flutter UI.
7. App-side domain and transport
- The Flutter models include StackChan/XiaoZhi concepts such as device, dance, expression, agent, conversation, registration, and music info.
- Network code is separated into HTTP and WebSocket utilities, indicating a split between request/response APIs and live sessions.
- Utility modules handle audio, Bluetooth, ML Kit, RSA, MAC validation, and native bridge calls, so the app also performs local device interaction and media handling.
8. Backend server as shared control plane
- server/ is a Go backend with a layered structure: API definitions, controllers, services, DAO/model layers, middleware, web socket support, and deployment manifests.
- It exposes versioned endpoints for admin, app store, dance, device, file, friend, pano, post, stackchan device, user, and xiaozhi.
- This server acts as the shared control plane for device registration/binding, content distribution, user accounts, and media/content management.
9. Server responsibilities and data persistence
- Controllers map HTTP APIs to services; services encapsulate business logic; DAOs handle storage access through generated/internal model types.
- The presence of sqlite_sequence entities and SQL setup scripts indicates relational persistence, while server/file/ and server/resource/ host static/media content.
- There is also WebSocket support in server/internal/web_socket/, likely for live device or avatar session updates.
10. Remote controller as a separate embedded client
- remote/code/ is another ESP-IDF/LVGL application for an ESP32-based handheld/standalone controller.
- It contains its own joystick handling, ESP-NOW initialization, and LVGL screens for setup, running, and IMU views.
- This firmware is a direct peer of the robot, focused on local control rather than cloud orchestration.
11. Primary data flows
- User ↔ Flutter app ↔ backend server: account/login, device binding, app store content, dance lists, files, and XiaoZhi token/license workflows.
- Flutter app ↔ robot firmware: live control, monitoring, video/avatar display, configuration, and OTA-related coordination.
- Remote controller ↔ robot firmware: low-latency ESP-NOW control for motion/interaction without going through the cloud.
- Robot firmware ↔ hardware: servos, LEDs, sensors, camera, audio, and touch are managed locally through HAL and motion subsystems.
12. Shared content and assets
- The repo includes common robot assets: 3D models (.glb, .scn), setup videos, icons, sounds, and UI artwork.
- These assets are consumed across platforms: Flutter UI, iOS native views, Android resources, and embedded firmware assets.
- This means the architecture has a “shared content” layer even though the runtimes differ.
13. Deployment boundary
- The backend includes Docker/Kustomize manifests and config, implying containerized deployment.
- The firmware side is built as embedded binaries with ESP-IDF and board-specific configuration.
- The mobile app is separately packaged for iOS and Android with native platform projects under app/ios and app/android.
14. What to draw in an architecture diagram
- Draw four main boxes: mobile app, robot firmware, remote controller firmware, and backend server.
- Show the server as the persistent control/content plane, the robot firmware as the hardware/runtime plane, the mobile app as the user-facing orchestration plane, and the remote controller as a direct low-latency input plane.
- Annotate the robot firmware with internal layers: HAL → stackchan behavior engine → app modes.
Main entry points: lib Flutter app · hal HAL layer · assets Shared media · main.cpp Main app
Components 21
Mobile app
Flutter client plus native iOS/Android integrations.
- Flutter app · Main cross-platform mobile client for setup, pairing, content, and live control.
app/lib - App transport · Network layer separating request APIs from live sessions.
app/lib/network - App models · Device, dance, expression, agent, conversation, registration, and music data models.
app/lib/model - iOS native · Native robot rendering and motion views plus bridge code for iOS.
app/ios/Runner - Android native · Android native activity, models, and 3D rendering assets.
app/android/app/src/main
Robot firmware
ESP32-S3 firmware that owns the physical robot.
- Main app · ESP-IDF entrypoint and runtime wiring for the robot firmware.
firmware/main/main.cpp - HAL layer · Board-specific hardware access for audio, BLE, ESP-NOW, network, OTA, RTC, servos, IMU, camera, display, touch, and glue.
firmware/main/hal - Behavior engine · Avatar, modifiers, motion, animation, JSON helpers, and utility math for robot behavior.
firmware/main/stackchan - Robot apps · User-facing firmware modes such as launcher, setup, app center, AI agent, avatar, dance, and EZ data.
firmware/main/apps - Firmware assets · Icons, fonts, sounds, and setup imagery bundled into the robot runtime.
firmware/main/assets
Remote controller
ESP32 handheld controller with local ESP-NOW input.
- Remote app · Main firmware for the handheld remote controller.
remote/code/main/StackChan-RemoteControl-ESPNow.cpp - ESP-NOW link · Low-latency peer-to-peer control initialization and communication.
remote/code/main/esp_now - Remote UI · Setup, running, and IMU views for the standalone controller.
remote/code/main/ui - Joystick input · Joystick handling and input mapping for motion control.
remote/code/main/joystick
Backend server
Go control plane for accounts, content, and device management.
- API surface · Admin, app store, dance, device, file, friend, pano, post, stackchan device, user, and xiaozhi APIs.
server/api - Controllers · Maps APIs to application services and request workflows.
server/internal/controller - Services · Domain logic for users, devices, apps, dances, files, and agents.
server/internal/service - Data layer · Persistence models and DAOs backed by relational storage and generated internal entities.
server/internal/dao - WebSocket hub · Socket support for live device or avatar session updates.
server/internal/web_socket - Deploy manifests · Docker and Kustomize deployment configuration.
server/manifest
Shared assets
Common media and 3D resources used across runtimes.
- Shared media · 3D models, setup videos, music, icons, and UI artwork used across app and firmware.
app/assets
Connections 26
- Flutter app → App transport: uses · Flutter UI uses transport utilities for backend and device communication.
- Flutter app → App models: binds · Screens and flows bind to product domain models.
- Flutter app → iOS native: bridges · iOS-specific rendering and motion views are launched from the app.
- Flutter app → Android native: bridges · Android native layer supports local robot visualization.
- App transport → API surface: HTTP/WebSocket · App talks to backend endpoints and live sessions.
- Flutter app → HAL layer: device control · App coordinates setup, binding, monitoring, and control with the robot.
- Main app → HAL layer: initializes · Entrypoint wires board services and hardware subsystems.
- Main app → Behavior engine: runs · Main runtime drives stackchan behavior state and actuation.
- Main app → Robot apps: hosts · Firmware exposes multiple user-facing modes and mini-apps.
- Robot apps → Behavior engine: invokes · App modes reuse avatar, motion, and modifier primitives.
- Behavior engine → HAL layer: acts on · Behavior layer ultimately drives servos, display, audio, camera, and sensors through HAL.
- HAL layer → Firmware assets: loads · Robot runtime consumes embedded icons, sounds, and setup art.
- Remote app → Joystick input: reads · Remote firmware polls joystick and local controls.
- Remote app → ESP-NOW link: sends · Main remote firmware communicates over ESP-NOW.
- Remote app → Remote UI: renders · Local UI shows setup, running, and IMU state.
- ESP-NOW link → HAL layer: peer control · Remote controller directly drives the robot via low-latency radio link.
- API surface → Controllers: dispatches · Versioned APIs are implemented by controllers.
- Controllers → Services: calls · Controllers delegate business logic to services.
- Services → Data layer: persists · Services read and write through DAOs and generated models.
- Controllers → WebSocket hub: upgrades · Some workflows use WebSocket sessions for live updates.
- API surface → Deploy manifests: deploys · Backend ships with container and Kustomize manifests.
- Shared media → Flutter app: consumed by · Flutter UI uses common art, video, and model assets.
- Shared media → iOS native: consumed by · iOS native views render shared 3D assets.
- Shared media → Android native: consumed by · Android native layer includes shared 3D model assets.
- Shared media → Firmware assets: mirrored in · Embedded runtime bundles a subset of shared content.
- Services → Main app: orchestrates · Backend supports registration, binding, content, and token workflows for devices.