* Add support for LCD Wiki ES3C35P (3.5" ESP32-S3) board
* fix: update variant counts in test for new ES3C35P board
---------
Co-authored-by: Eivs <eivs@Eivss-MacBook-Pro.local>
- Updated BoxAudioCodec constructor to include parameters for input gain, reference gain channel, and reference gain.
- Modified EnableInput method to set gain for the reference channel if applicable.
- Adjusted CustomAudioCodec initialization to utilize new gain parameters for improved audio processing.
- Added PressToTalkMcpTool to manage press-to-talk and click-to-talk modes.
- Updated button event handling to support listening state transitions based on press-to-talk status.
- Initialized press-to-talk tool during board setup for enhanced user interaction.
* feat: migrate firmware to xiaozhi-fonts 2.0.0
* fix: restart scrolling subtitles on text change
* docs: add AGENTS.md for project overview, architecture, rules, and commands
- Introduced AGENTS.md to document the XiaoZhi voice-assistant firmware architecture, required rules for development, board configuration process, and validation steps.
- Included detailed sections on project structure, commands for building and testing, and authoritative documentation references.
* feat: finalize glyph push protocol
* fix: remove stale emoji font include
* build: use xiaozhi-fonts 2.0.0
* fix: decouple custom fonts from glyph push
* refactor: improve wake word invocation handling
- Introduced BeginWakeWordInvoke method to streamline the wake word processing flow.
- Updated HandleWakeWordDetectedEvent to utilize the new method, ensuring proper state transitions and scheduling.
- Enhanced error handling to prevent the device from getting stuck in the connecting state.
- Refactored audio channel management to improve responsiveness during wake word detection.
---------
Co-authored-by: Xiaoxia <terrence.huang@tenclass.com>
* Migrate board builds to ESP-IDF 6.0.1
* Document upstream IDF 6 CI blockers
* Use IDF 6 patched registry components
* Fix board-specific ESP-IDF 6 build errors
* Fix remaining ESP-IDF 6 board builds
* Document final ESP-IDF 6 matrix results [skip ci]
* update IDF 6 component releases
* rebase IDF 6 migration and use upstream SSCMA
* Enhance audio input management in AudioService
- Introduced AS_EVENT_AUDIO_INPUT_STOP_REQUEST to manage audio input stopping more effectively.
- Updated AudioService::Start() to clear the new stop request event.
- Modified AudioService::AudioInputTask() to handle the stop request and ensure proper input disabling.
- Adjusted AdcPdmAudioCodec::Start() to reflect lazy input opening, ensuring start/stop operations run in the same task.
* Refactor audio codec management and configuration
- Added output_device_opened_ flag to track the state of the output device in Es8388 and Es8389 codecs.
- Updated EnableOutput method to prevent unnecessary device reopening and added mute functionality to manage audio output more effectively.
- Enhanced error handling in Es8311AudioCodec by ensuring proper closure and deletion of the codec device.
- Adjusted sample rates in board configurations for atk-dnesp32s3-box2 to 24000 Hz and introduced AUDIO_CODEC_USE_MCLK for improved clock management.
* Update build configurations and documentation for ESP-IDF 6 compatibility
- Added container specification for the build workflow using espressif/idf:v6.0.1.
- Updated the version of the espressif/esp_video component to ^2.3.0.
- Enhanced documentation to clarify the support status of ESP32-P4 variants, specifying that legacy Rev < 3 variants are excluded from the IDF 6 matrix.
- Adjusted board configurations to include IDF version constraints and necessary SDK configurations for legacy support.
- Improved handling of YUV formats in EspVideo to maintain compatibility with previous versions.
* Implement playback drained event handling in Application
- Added MAIN_EVENT_PLAYBACK_DRAINED to manage playback state transitions.
- Introduced callbacks for playback drained events in AudioService to trigger listening state changes.
- Refactored Application::Run() to handle deferred listening start based on playback queue status.
- Enhanced audio processing logic to prevent audio truncation during state changes.
- Updated related methods to ensure proper wake word detection configuration during listening mode.
* Fix variant selection shell in CI
* Update project version to 2.4.0 and adjust component dependencies
- Bump project version in CMakeLists.txt to 2.4.0.
- Change espressif/esp_video component version to ^2.0.1 in idf_component.yml.
- Modify AUDIO_INPUT_REFERENCE setting in config.h to false for m5stack-core-s3.
- Remove unnecessary infinite loops in xmini_c3_board.cc during initialization.
---------
Co-authored-by: Xiaoxia <terrence.huang@tenclass.com>
- Implemented a new function, IsEs8311Present(), to verify the presence of the ES8311 codec by checking its chip ID registers instead of relying solely on I2C bus probing.
- Updated error logging to provide clearer messages when the ES8311 codec is not detected.
- Added a continuous logging loop to indicate waiting for the boot button to be pressed during initialization.
* Update QQ group contact in README.md
* Update dependencies in idf_component.yml and modify AFE audio processor configuration
- Updated versions for several dependencies in idf_component.yml to improve compatibility and performance.
- Changed AFE audio processor initialization to use low-cost mode and adjusted AEC settings for enhanced audio processing.
* Update audio dependencies in idf_component.yml
- Downgraded versions for espressif/esp_audio_effects and espressif/esp_audio_codec to improve compatibility with existing components.
* refactor(audio): unify speech processing engines
---------
Co-authored-by: Xiaoxia <terrence.huang@tenclass.com>
- Renamed CONFIG_XIAOZHI_USE_ETHERNET to CONFIG_XIAOZHI_NETWORK_ETHERNET for clarity.
- Updated CMakeLists.txt and Kconfig.projbuild to reflect the new configuration option.
- Adjusted conditional compilation in system_info.cc and esp32-p4-nano.cc to use the new network configuration.
- Added a rename file to maintain compatibility with existing sdkconfig files.
* Enable WebSocket control server for electron-bot
* Add servo control features to ElectronBot
- Introduced new actions for servo movement and sequences, including ACTION_SERVO_MOVE and ACTION_SERVO_SEQUENCE.
- Implemented methods for clamping servo positions and applying oscillation effects.
- Added GetServoPositions method to retrieve current servo angles.
- Enhanced README.md with detailed AI command examples and servo capabilities.
These changes improve the flexibility and control of the ElectronBot's movements, allowing for more complex actions and better integration with AI functionalities.
* Refactor servo control in ElectronBot
- Removed unnecessary home action logic during servo sequences to improve action fluidity.
- Updated README.md to clarify the action parameters for hand movements, specifically the amplitude for flapping actions.
- Introduced clamping functions for servo angles and amplitudes to ensure safe operation within defined limits.
These changes enhance the control and safety of the ElectronBot's servo movements, allowing for more precise and natural actions.
* Enhance Otto Robot camera support by adding configuration for OV3660. Updated config.h to define camera types and GPIO settings, modified config.json to include new camera options, and refactored otto_robot.cc for improved camera detection and initialization logic.
* fix: 移除 OttoEmojiDisplay 构造函数中的 SetTheme 调用以修复 LoadProhibited 崩溃
Made-with: Cursor
* refactor: improve audio service error handling and codec timeout management
- Updated AudioService to prevent input task termination on read timeout, introducing a delay instead.
- Enhanced NoAudioCodec to implement a read timeout for I2S channel reads.
- Adjusted WebSocketControlServer to set a control port for improved socket management.
- Added manufacturer information to the config.json for waveshare ESP32-Touch-LCD-3.5.
* fix(otto): WebSocket direct clients not receiving MCP responses
When a browser connects directly to the WebSocket control server (port
8080) and sends a JSON-RPC request, the MCP response was routed through
Application::SendMcpMessage -> protocol_->SendMcpMessage, which sends it
to the cloud protocol channel. As a result, the direct WebSocket client
never received the response, while the WeChat mini-program could because
it communicates via the cloud.
Fix:
- Add BroadcastMessage() to WebSocketControlServer, using
httpd_queue_work + httpd_ws_send_frame_async to asynchronously
send responses back to all connected clients on port 8080
- Add RegisterMcpBroadcastCallback() to Application, allowing an
additional MCP send callback to be registered; SendMcpMessage()
now invokes it alongside the cloud protocol
- Register the broadcast callback in OttoRobot after the WebSocket
server starts successfully
Also add WebSocket direct-connect API documentation to README.md
with complete JSON-RPC 2.0 command examples.
* fix(otto-robot): migrate camera backend and set safe dark default theme
- Migrate `otto-robot` camera backend from `EspVideo` to `Esp32Camera` to improve capture stability after reboot/power cycle.
- Keep runtime sensor detection for both OV2640 and OV3660, and rename PID macros with `OTTO_` prefix to avoid symbol conflicts.
- Configure camera output as `RGB565 + FRAMESIZE_240X240` to match the 240x240 display.
- Rotate OV2640 output by 180 degrees (`VFlip + HMirror`) for correct orientation.
- Simplify `otto-robot` camera sdkconfig options by keeping only:
- `CONFIG_CAMERA_OV2640=y`
- `CONFIG_CAMERA_OV3660=y`
- Move Otto default dark theme setup into `OttoEmojiDisplay::SetupUI()` (after base UI init) to avoid boot-time LVGL null-object crash caused by calling `SetTheme()` too early.
* feat(motion): smooth action ending and adaptive homing on otto/electron-bot
Improve servo motion transitions to reduce abrupt returns to home pose.
Replace linear interpolation with ease-out movement, make homing duration adaptive to angle delta, and skip intermediate homing when queued actions are pending to keep multi-action sequences fluid.
* fix(electron-bot): show chat subtitle on display
ElectronEmojiDisplay overrode SetupChatLabel() to delete the parent's
chat_message_label_ (which lives inside bottom_bar_) and recreate it on the
bottom-layer container_ without alignment. As a result the real subtitle label
was hidden behind emoji_box_/top_bar_, while SetChatMessage() only toggled the
now-empty bottom_bar_, so chat subtitles were never visible.
Reuse the parent label created in bottom_bar_ (matching otto-robot) and only
switch to the dark theme in SetupUI(), removing the custom SetupChatLabel().
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(electron-bot): restore status icons on standby and clean up display
- SetStatus() did not re-show network_label_/battery_label_ when returning
to STANDBY, so the WiFi and battery icons stayed hidden after listening or
speaking. Restore them on standby to match otto-robot.
- Remove the unused InitializeElectronEmojis() method and its declaration.
- Drop unused includes (vector, assets.h, emoji_collection.h, lvgl_image.h)
and add the missing trailing newline in the header.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(esp32-s3-touch-amoled-1.8): keep wake word always-on while charging
The previous logic in GetBatteryLevel() used a static last_discharging
initialized to false. If the device booted on the charger (discharging
already false), the equality check never fired and PowerSaveTimer kept
running, so the device entered light sleep after 60s and stopped wake
word detection — defeating the always-on-while-charging behavior.
Move the tracker to a member with a tri-state initial value (-1 =
unknown) so the very first GetBatteryLevel() tick reconciles the timer
state regardless of boot conditions. Add ESP_LOGI on each transition
so the always-on state is visible in the logs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(esp32-s3-touch-amoled-1.8): keep PowerSaveTimer ticking while charging
The previous patch disabled PowerSaveTimer entirely on the charger,
which also disabled the screen dim and sleep emoji that fire from
OnEnterSleepMode after 60s of idle. Since this board constructs the
timer with cpu_max_freq=-1, it never enters CPU light sleep nor
disables wake word detection in OnEnterSleepMode anyway — the only
charger-incompatible action is the 5-minute pmic_->PowerOff() in
OnShutdownRequest.
Keep the timer running unconditionally and gate just the shutdown
callback on last_discharging_. Also call WakeUp() on the
charging→battery transition so an unplug after 5+ idle minutes
doesn't trigger an immediate power-off.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* Revert "fix(esp32-s3-touch-amoled-1.8): keep PowerSaveTimer ticking while charging"
This reverts commit d7a9b619ebfd2bc1573865fa2f85e8d636625718.
* feat(esp32-s3-touch-amoled-1.8): add wake word heartbeat while charging
Brings back the display dim + sleepy emoji UX while plugged in by
keeping PowerSaveTimer running and gating just the shutdown callback
on charging state. To work around the wake word stalling after long
idle (~22 min observed in testing), add a periodic bounce timer that
restarts wake word detection every 5 minutes while idle on the
charger. Skips the bounce mid-conversation, on battery, or when wake
word is already disabled.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(esp32-s3-touch-amoled-1.8): bounce wake word on sleep mode entry
Hardware testing shows wake word detection stops responding shortly
after PowerSaveTimer enters sleep mode (display dim + sleepy emoji).
The 5-minute periodic heartbeat doesn't help — the failure happens
well before the first bounce.
Add a preemptive bounce in OnEnterSleepMode while charging, and
shorten the periodic heartbeat to 2 minutes as a safety net for any
later stalls.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* fix(esp32-s3-touch-amoled-1.8): use standalone dim timer on charger
Hardware testing showed that the previous approach (keeping
PowerSaveTimer active with wake-word bounces on the charger) still
left wake word unresponsive after sleep mode entered — even with a
preemptive bounce in OnEnterSleepMode and a 2-min periodic heartbeat,
the AFE / audio pipeline kept stalling, and the user had to wake the
device with the boot button.
Switch to a different split:
- On charger: PowerSaveTimer is fully disabled. Its sleep callbacks
never fire, so wake word detection is undisturbed. A standalone
dim_timer (1s tick) handles the brightness + sleepy-emoji UX
independently — on idle 60s it dims, on activity (conversation,
button) it restores via Application::CanEnterSleepMode() polling.
- On battery: PowerSaveTimer runs as before (60s sleep callbacks,
5min auto-shutdown), dim_timer stays stopped.
Also drops the wake-word bounce timer entirely — without
PowerSaveTimer's sleep mode firing on the charger there's nothing to
recover from.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
* Enhance Otto Robot camera support by adding configuration for OV3660. Updated config.h to define camera types and GPIO settings, modified config.json to include new camera options, and refactored otto_robot.cc for improved camera detection and initialization logic.
* fix: 移除 OttoEmojiDisplay 构造函数中的 SetTheme 调用以修复 LoadProhibited 崩溃
Made-with: Cursor
* refactor: improve audio service error handling and codec timeout management
- Updated AudioService to prevent input task termination on read timeout, introducing a delay instead.
- Enhanced NoAudioCodec to implement a read timeout for I2S channel reads.
- Adjusted WebSocketControlServer to set a control port for improved socket management.
- Added manufacturer information to the config.json for waveshare ESP32-Touch-LCD-3.5.
* fix(otto): WebSocket direct clients not receiving MCP responses
When a browser connects directly to the WebSocket control server (port
8080) and sends a JSON-RPC request, the MCP response was routed through
Application::SendMcpMessage -> protocol_->SendMcpMessage, which sends it
to the cloud protocol channel. As a result, the direct WebSocket client
never received the response, while the WeChat mini-program could because
it communicates via the cloud.
Fix:
- Add BroadcastMessage() to WebSocketControlServer, using
httpd_queue_work + httpd_ws_send_frame_async to asynchronously
send responses back to all connected clients on port 8080
- Add RegisterMcpBroadcastCallback() to Application, allowing an
additional MCP send callback to be registered; SendMcpMessage()
now invokes it alongside the cloud protocol
- Register the broadcast callback in OttoRobot after the WebSocket
server starts successfully
Also add WebSocket direct-connect API documentation to README.md
with complete JSON-RPC 2.0 command examples.
* fix(otto-robot): migrate camera backend and set safe dark default theme
- Migrate `otto-robot` camera backend from `EspVideo` to `Esp32Camera` to improve capture stability after reboot/power cycle.
- Keep runtime sensor detection for both OV2640 and OV3660, and rename PID macros with `OTTO_` prefix to avoid symbol conflicts.
- Configure camera output as `RGB565 + FRAMESIZE_240X240` to match the 240x240 display.
- Rotate OV2640 output by 180 degrees (`VFlip + HMirror`) for correct orientation.
- Simplify `otto-robot` camera sdkconfig options by keeping only:
- `CONFIG_CAMERA_OV2640=y`
- `CONFIG_CAMERA_OV3660=y`
- Move Otto default dark theme setup into `OttoEmojiDisplay::SetupUI()` (after base UI init) to avoid boot-time LVGL null-object crash caused by calling `SetTheme()` too early.
* feat(motion): smooth action ending and adaptive homing on otto/electron-bot
Improve servo motion transitions to reduce abrupt returns to home pose.
Replace linear interpolation with ease-out movement, make homing duration adaptive to angle delta, and skip intermediate homing when queued actions are pending to keep multi-action sequences fluid.
* Enhance Otto Robot camera support by adding configuration for OV3660. Updated config.h to define camera types and GPIO settings, modified config.json to include new camera options, and refactored otto_robot.cc for improved camera detection and initialization logic.
* fix: 移除 OttoEmojiDisplay 构造函数中的 SetTheme 调用以修复 LoadProhibited 崩溃
Made-with: Cursor
* refactor: improve audio service error handling and codec timeout management
- Updated AudioService to prevent input task termination on read timeout, introducing a delay instead.
- Enhanced NoAudioCodec to implement a read timeout for I2S channel reads.
- Adjusted WebSocketControlServer to set a control port for improved socket management.
- Added manufacturer information to the config.json for waveshare ESP32-Touch-LCD-3.5.
* fix(otto): WebSocket direct clients not receiving MCP responses
When a browser connects directly to the WebSocket control server (port
8080) and sends a JSON-RPC request, the MCP response was routed through
Application::SendMcpMessage -> protocol_->SendMcpMessage, which sends it
to the cloud protocol channel. As a result, the direct WebSocket client
never received the response, while the WeChat mini-program could because
it communicates via the cloud.
Fix:
- Add BroadcastMessage() to WebSocketControlServer, using
httpd_queue_work + httpd_ws_send_frame_async to asynchronously
send responses back to all connected clients on port 8080
- Add RegisterMcpBroadcastCallback() to Application, allowing an
additional MCP send callback to be registered; SendMcpMessage()
now invokes it alongside the cloud protocol
- Register the broadcast callback in OttoRobot after the WebSocket
server starts successfully
Also add WebSocket direct-connect API documentation to README.md
with complete JSON-RPC 2.0 command examples.
* fix(otto-robot): migrate camera backend and set safe dark default theme
- Migrate `otto-robot` camera backend from `EspVideo` to `Esp32Camera` to improve capture stability after reboot/power cycle.
- Keep runtime sensor detection for both OV2640 and OV3660, and rename PID macros with `OTTO_` prefix to avoid symbol conflicts.
- Configure camera output as `RGB565 + FRAMESIZE_240X240` to match the 240x240 display.
- Rotate OV2640 output by 180 degrees (`VFlip + HMirror`) for correct orientation.
- Simplify `otto-robot` camera sdkconfig options by keeping only:
- `CONFIG_CAMERA_OV2640=y`
- `CONFIG_CAMERA_OV3660=y`
- Move Otto default dark theme setup into `OttoEmojiDisplay::SetupUI()` (after base UI init) to avoid boot-time LVGL null-object crash caused by calling `SetTheme()` too early.
* perf(websocket): switch WiFi to performance mode before connecting
Optimize WebSocket connection speed by switching WiFi to performance
mode before establishing the connection, instead of after.
This reduces network latency significantly:
- TCP connection: 1093ms → 88ms (92% faster)
- WebSocket handshake: 1035ms → 80ms (92% faster)
- Total network layer: 2128ms → 173ms (92% faster)
The issue was caused by WiFi power save mode (MAX_MODEM) which adds
significant latency to packet transmission.
* Adjust formatting
* Enhance Otto Robot camera support by adding configuration for OV3660. Updated config.h to define camera types and GPIO settings, modified config.json to include new camera options, and refactored otto_robot.cc for improved camera detection and initialization logic.
* fix: 移除 OttoEmojiDisplay 构造函数中的 SetTheme 调用以修复 LoadProhibited 崩溃
Made-with: Cursor
* refactor: improve audio service error handling and codec timeout management
- Updated AudioService to prevent input task termination on read timeout, introducing a delay instead.
- Enhanced NoAudioCodec to implement a read timeout for I2S channel reads.
- Adjusted WebSocketControlServer to set a control port for improved socket management.
- Added manufacturer information to the config.json for waveshare ESP32-Touch-LCD-3.5.
* fix(otto): WebSocket direct clients not receiving MCP responses
When a browser connects directly to the WebSocket control server (port
8080) and sends a JSON-RPC request, the MCP response was routed through
Application::SendMcpMessage -> protocol_->SendMcpMessage, which sends it
to the cloud protocol channel. As a result, the direct WebSocket client
never received the response, while the WeChat mini-program could because
it communicates via the cloud.
Fix:
- Add BroadcastMessage() to WebSocketControlServer, using
httpd_queue_work + httpd_ws_send_frame_async to asynchronously
send responses back to all connected clients on port 8080
- Add RegisterMcpBroadcastCallback() to Application, allowing an
additional MCP send callback to be registered; SendMcpMessage()
now invokes it alongside the cloud protocol
- Register the broadcast callback in OttoRobot after the WebSocket
server starts successfully
Also add WebSocket direct-connect API documentation to README.md
with complete JSON-RPC 2.0 command examples.
The guard around the registration of the self.assets.set_download_url tool has been removed, ensuring it is always available for configuration. This change addresses issues on 32MB flash devices where the tool was previously skipped due to partition validation checks.
Fixes#1962
* fix(m5stack-tab5): remove stale esp_video==0.7.0 dependency instructions
The README previously instructed users to override esp_video to 0.7.0
and esp_ipa to 0.1.0, but this causes build failures because:
- esp_video 0.7.0 does not export esp_video_deinit(), resulting in
linker errors ('MAP_FAILED' and 'esp_video_deinit' not declared)
- The project's main/idf_component.yml already pins the correct
version (esp_video==1.3.1) that the source code expects
Users should now use the default dependency versions from idf_component.yml
without modification.
Fixes#1957
* fix(mcp): always register self.assets.set_download_url tool
On 32MB flash devices the assets partition layout differs from the
default, causing partition_valid() to return false and silently
skipping registration of the self.assets.set_download_url MCP tool.
Users see 'Unknown tool: self.assets.set_download_url' from their MCP
client.
The tool writes to Settings storage which works regardless of the
partition map, so the partition_valid() guard is unnecessary.
Move the AddUserOnlyTool call outside the guard so the tool is always
available for explicit configuration via MCP.
Fixes#1962
---------
Co-authored-by: Aayush Pratap Singh <aayushpratap.singh@gmail.com>
Create main/boards/rymcu/bigsmart so future RYMCU boards can live under the same manufacturer directory. Update CMake to set MANUFACTURER to rymcu while preserving BOARD_NAME as rymcu-bigsmart, and adjust config.json so release output remains rymcu-bigsmart.
* fix(blufi): GET_WIFI_LIST triggers real-time scan with guaranteed response
Previously, ESP_BLUFI_EVENT_GET_WIFI_LIST waited for any in-progress scan
to finish and then returned the cached result. When the cache was empty
(e.g. after a config-mode transition that stopped the Wi-Fi driver),
_send_wifi_list() returned silently with no response frame, leaving the
App waiting until timeout.
Changes:
- GET_WIFI_LIST now clears the cache and starts a fresh scan immediately.
- _wifi_scan_event_handler calls _send_wifi_list() after every scan
triggered by a GET_WIFI_LIST request.
- start_wifi_scan() calls esp_wifi_start() before esp_wifi_scan_start()
to handle the case where the driver was stopped during a mode
transition (ESP_ERR_WIFI_STATE is treated as already-started).
- _send_wifi_list() sends ESP_BLUFI_WIFI_SCAN_FAIL when no APs are
found, so the App always receives a terminal response.
- Redundant static_cast in _wifi_scan_event_handler replaced with the
existing local `self` pointer.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(blufi): preserve cache fast-path, fall back to live scan only when needed
Address review feedback on always-rescan latency regression.
The GET_WIFI_LIST handler now distinguishes three cases:
1. Scan in flight: defer the response via m_send_list_after_scan; the
scan-done handler dispatches when it fires. Removes the previous
blocking `while (m_scan_in_progress) vTaskDelay(500)` which would
stall the BluFi event task indefinitely if the scan never completed.
2. Cache populated: respond from cache immediately (~50 ms, no latency
change vs original behavior). _send_wifi_list() still kicks off an
async refresh scan as before to keep the cache fresh.
3. Cache empty and no scan running: trigger a live scan and dispatch
from the scan-done handler. If start_wifi_scan() fails, send
ESP_BLUFI_WIFI_SCAN_FAIL so the App exits its wait state.
State variables are also disentangled:
- m_scan_should_save_ssid keeps its original meaning (write scan results
into m_ap_records). Cleared during connect-to-AP so the connect-time
scan does not pollute the cache.
- m_send_list_after_scan is new and tracks "the next scan-done event
should respond to a pending GET_WIFI_LIST request". The previous PR
conflated these two responsibilities onto m_scan_should_save_ssid,
which would have caused init-time scans to spuriously emit a wifi list
to the App.
start_wifi_scan() now returns bool so the caller can distinguish
"scan started or already running" from "could not start a scan".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Yixin Shi <shiyixin@qiniu.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
【中文】
7B 变体使用 v1.x 版本的 ESP32-P4 芯片和 32MB flash,但 config.json 中
对应的 sdkconfig_append 既没声明芯片版本,也没声明 flash 大小。IDF 因此
默认按 rev v3.1 + 16MB flash 构建,bootloader 被链接脚本
bootloader.rev3.ld 放到 0x4ffa_xxxx 的高位 HP_SRAM,这个区域在 v1.x
硅上不可取指,ROM 加载完跳过去,CPU 在第一条指令就
"Guru Meditation Error: Core 0 panic'ed (Illegal instruction)",
bootloader log 一行都打不出来。
添加 CONFIG_ESP32P4_SELECTS_REV_LESS_V3=y + CONFIG_ESP32P4_REV_MIN_100=y
让 bootloader.ld(低位 HP_SRAM 0x4ff2_xxxx)被选中,并声明
CONFIG_ESPTOOLPY_FLASHSIZE_32MB=y + partitions/v2/32m.csv,使 image
header 和分区表和板子实际硬件对齐;CONFIG_ESPTOOLPY_FLASH_MODE_AUTO_DETECT=n
与已知 32MB 板子 sensecap-watcher 保持一致。
在 v1.3 版本的 P4-WIFI6-Touch-LCD-7B 上验证通过:bootloader 段现在加载到
0x4ff33ce0 / 0x4ff29ed0 / 0x4ff2cbd0;启动过程中 MIPI-DSI LCD、GT911
触控、OV5647 摄像头、ES8311+ES7210 音频和 ESP32-C6 SDIO 协处理器
全部正常初始化;状态机进入 wifi_configuring,对外发出 Xiaozhi-XXXX 配网热点。
---
[English]
The 7B variant ships with a v1.x ESP32-P4 silicon and a 32 MB flash, but
the existing sdkconfig_append declared neither. IDF then defaults to rev
v3.1 + 16 MB, and the bootloader is linked via bootloader.rev3.ld into
the 0x4ffa_xxxx upper HP_SRAM window. That range is not fetchable on
v1.x silicon, so the ROM jumps to the loaded bootloader and the CPU
panics on the first opcode with "Guru Meditation Error: Core 0 panic'ed
(Illegal instruction)" before any bootloader log is printed.
Add CONFIG_ESP32P4_SELECTS_REV_LESS_V3=y + CONFIG_ESP32P4_REV_MIN_100=y
so bootloader.ld (low HP_SRAM 0x4ff2_xxxx) is used, and declare
CONFIG_ESPTOOLPY_FLASHSIZE_32MB=y + partitions/v2/32m.csv so the image
header and partition offsets match the board. Disable flash-mode
auto-detect for parity with the other known 32 MB board (sensecap-watcher).
Verified on a v1.3 P4-WIFI6-Touch-LCD-7B: bootloader segments now load
at 0x4ff33ce0 / 0x4ff29ed0 / 0x4ff2cbd0; boot completes through MIPI DSI,
GT911 touch, OV5647 camera, ES8311+ES7210 audio and the ESP32-C6 SDIO
coprocessor; device reaches wifi_configuring and exposes the Xiaozhi-XXXX AP.
- Added a comprehensive guide for creating custom boards in the XiaoZhi AI project, detailing directory structure, configuration files, and initialization code.
- Introduced a new document explaining the MCP protocol for IoT control, including message formats and interaction flows.
- Updated existing documentation to reflect changes in tool registration and usage examples for the MCP protocol.
- Enhanced README files for better clarity and consistency across languages.
Add full keyboard support and keyboard-based WiFi configuration for
M5Stack Cardputer Adv:
- TCA8418 I2C keyboard driver with 56-key matrix, interrupt-driven
key events, and debounce handling
- Keyboard WiFi config UI: scan/select/input SSID and password
directly on the device without needing a phone
- Volume control (up/down arrows) and brightness control (left/right)
via keyboard with fine-step adjustment near bounds
- Enter key to toggle chat state
- Display offset and backlight fixes for ST7789V2
- README with flash parameters and hardware specs
Co-authored-by: bot <bot@localhost>
* Initial plan
* Fix crash when adjusting volume via button while audio device is disabled
When both input and output are disabled, UpdateDeviceState() sets dev_ to
nullptr. Pressing a volume button calls SetOutputVolume() which previously
called esp_codec_dev_set_out_vol(dev_, volume) without null-checking dev_,
causing a crash via ESP_ERROR_CHECK.
Fix: Add null guard for dev_ and mutex lock for thread safety. The volume
is still saved via AudioCodec::SetOutputVolume() and will be applied when
the device is reopened by UpdateDeviceState().
Agent-Logs-Url: https://github.com/78/xiaozhi-esp32/sessions/945c653a-ed16-49af-aefe-5cfb473402c6
Co-authored-by: 78 <4488133+78@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: 78 <4488133+78@users.noreply.github.com>
* Add touch screen support to freenove 2.8
Volume and brightness levels on vertical/horizontal swipe
Listening, chat state and wifi config on press/long press
* Simplified
Removed swipes. Touch quiet unresponsive on Freenove board.
* Remove unused parts
* Modify long tap duration in TouchTask
Updated touch driver logic to change long tap duration from 600ms to 3000ms.
* Cosmetic fix
* Update dependencies in idf_component.yml to latest versions: esp-ml307 to ~3.6.5, uart-eth-modem to ~0.3.4, esp32-camera to ^2.1.5, lvgl to ~9.5.0, esp_lvgl_port to ~2.7.2, esp_mmap_assets to ^1.3.2, and adc_battery_estimation to ^0.2.1.
* Update esp-wifi-connect version and adjust partition sizes in 4m.csv
* Initial plan
* Add GPIO_NUM_NC guards for LampController and SingleLed across all board files
Boards crash when LAMP_GPIO or BUILTIN_LED_GPIO is set to GPIO_NUM_NC because
LampController and SingleLed are instantiated unconditionally.
Fix: Guard LampController with `if (LAMP_GPIO != GPIO_NUM_NC)` in InitializeTools()
and guard SingleLed with `if (BUILTIN_LED_GPIO != GPIO_NUM_NC)` in GetLed(),
falling back to NoLed when GPIO is not connected.
Fixes the reported crash on bread-compact-wifi-lcd and hardens all similar boards.
Co-authored-by: 78 <4488133+78@users.noreply.github.com>
* Fix crash when LAMP_GPIO or BUILTIN_LED_GPIO is GPIO_NUM_NC (minimal 2-file fix)
Instead of adding guards in every board file (36 files), fix the root cause
in just 2 source files:
1. SingleLed constructor: replace assert(gpio != GPIO_NUM_NC) with graceful
early return, leaving led_strip_ as nullptr (all methods already null-check it)
2. LampController constructor: add early return when gpio_num == GPIO_NUM_NC
This fixes the crash on bread-compact-wifi-lcd and any other board where
these GPIOs are set to GPIO_NUM_NC.
Co-authored-by: 78 <4488133+78@users.noreply.github.com>
* Remove esp_timer_delete from destructor to match original behavior
Co-authored-by: 78 <4488133+78@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: 78 <4488133+78@users.noreply.github.com>
* Initial plan
* Fix GIF emotion switching crash: hide emoji_image_ before destroying gif_controller_
The crash was caused by a race condition in SetEmotion() where two separate
DisplayLockGuard scopes allowed the LVGL render task to access freed GIF image
data between lock scopes. By hiding emoji_image_ before destroying
gif_controller_ in the same lock scope, LVGL won't try to render the image
with freed data during the gap.
Co-authored-by: 78 <4488133+78@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: 78 <4488133+78@users.noreply.github.com>
On Windows, Python's default encoding is the system locale (e.g., cp1252,
gbk) rather than UTF-8. This causes UnicodeDecodeError when reading
sdkconfig files, CMakeLists.txt, or JSON configs that contain non-ASCII
characters (e.g., Chinese comments, UTF-8 BOM).
Fix: Add explicit encoding='utf-8' to all text file open() calls in:
- scripts/release.py (Path.open() for config.json, CMakeLists.txt, sdkconfig)
- scripts/build_default_assets.py (io.open() for sdkconfig, open() for headers/configs)
- scripts/versions.py (open() for info.json read/write)
Relates to #1792
Co-authored-by: Nicola Spieser <redbasecap@users.noreply.github.com>
* fix(echoear): Fix spelling errors in PCB version detection
- Fixed 'pcb_verison' -> 'pcb_version' (variable and log messages)
- Fixed 'Initializest77916Display' -> 'InitializeSt77916Display' (function name)
- Fixed 'PCB verison' -> 'PCB version' in log messages
These are cosmetic fixes that improve code readability and maintainability.
* fix(echoear): Add destructor for resource cleanup
- Add destructor to clean up tasks, objects, GPIO ISR, and temperature sensor
- Store task handles to allow proper task deletion
- Skip deleting backlight_ and camera_ as their base classes lack virtual destructors
- Improves code robustness without altering functional behavior
* add: Moji 2 has built-in ESP32-C5 dual band Wi-Fi
* feat: PowerSaveTimer 160 >> 240
* fix(audio): add frame buffering to NoAudioProcessor
On chips where CONFIG_USE_AUDIO_PROCESSOR is unavailable (e.g., ESP32-C5,
which is excluded by Kconfig despite having PSRAM), NoAudioProcessor is used.
The original implementation passed through 160-sample chunks directly, but
the Opus encoder expects 960 samples (60ms @ 16kHz), resulting in frame size
mismatch errors, high CPU usage, and watchdog resets.
Added output_buffer_ to accumulate audio samples until reaching
frame_samples_ (960) before outputting, matching AfeAudioProcessor behavior.
other:
Turn off sleep mode (PowerSaveTimer)
* fix(config): enable SPIRAM for ESP32-C5 build
* Enhance Otto Robot camera support by adding configuration for OV3660. Updated config.h to define camera types and GPIO settings, modified config.json to include new camera options, and refactored otto_robot.cc for improved camera detection and initialization logic.
* fix: 移除 OttoEmojiDisplay 构造函数中的 SetTheme 调用以修复 LoadProhibited 崩溃
Made-with: Cursor
* refactor: improve audio service error handling and codec timeout management
- Updated AudioService to prevent input task termination on read timeout, introducing a delay instead.
- Enhanced NoAudioCodec to implement a read timeout for I2S channel reads.
- Adjusted WebSocketControlServer to set a control port for improved socket management.
- Added manufacturer information to the config.json for waveshare ESP32-Touch-LCD-3.5.
* Enhance Otto Robot camera support by adding configuration for OV3660. Updated config.h to define camera types and GPIO settings, modified config.json to include new camera options, and refactored otto_robot.cc for improved camera detection and initialization logic.
* fix: 移除 OttoEmojiDisplay 构造函数中的 SetTheme 调用以修复 LoadProhibited 崩溃
Made-with: Cursor
* Enhance GitHub Actions artifact download script
- Updated the output directory structure to save downloaded files in a version-specific subdirectory (releases/<version>).
- Added a new function to determine the default releases directory path relative to the script's location.
- Improved artifact renaming logic to handle known extensions more robustly and ensure compatibility with filenames containing dots.
* Refactor UI setup in ElectronEmojiDisplay and OttoEmojiDisplay classes
- Moved SetupChatLabel call in ElectronEmojiDisplay to ensure it is executed after the parent UI is initialized, preventing potential issues with container validity.
- Updated SetupUI in OttoEmojiDisplay to release the display lock before calling SetEmotion, avoiding deadlock scenarios during UI setup.
* Add multiline chat message support in display configuration
- Introduced a new Kconfig option to enable multiline chat message display in the default mode.
- Updated the LCD display setup to accommodate a dynamic height bottom bar for multiline messages.
- Modified the configuration files for the waveshare-esp32-s3-epaper-1.54 board to include the new chat message setting.
* Update font and emoji settings for Magiclick boards; enhance bottom bar visibility logic in LCD display
- Changed the default text and emoji fonts for Magiclick S3 2P4 and S3 2P5 boards to Noto fonts.
- Improved bottom bar visibility logic in LcdDisplay to hide when there is no content, ensuring a cleaner UI experience.
* Add support for both hardware versions of waveshare-s3-epaper-1.54
* use v2 partition, reduced assets partition by 300k so that the app would fit
* add manufacturer
* updated readme
* CONFIG_SR_WN_WN9_NIHAOXIAOZHI_TTS=n to save space
* no custom font
* chore: Bump project version to 2.2.3 and refactor board configuration parsing in release script
- Updated PROJECT_VER in CMakeLists.txt from 2.2.2 to 2.2.3.
- Refactored the board configuration parsing logic in release.py to improve clarity and efficiency by eliminating the mapping function and directly searching for the board type in the CMakeLists.txt file.
* refactor: Remove unused system_info.h include and delete obsolete source root file
- Removed the inclusion of system_info.h in main.cc as it was no longer needed.
- Deleted the _codeql_detected_source_root file, which was obsolete.
* feat: Add lcd_init_cmds.h include for display initialization in esp32-p4-nano board
- Included the lcd_init_cmds.h header file in esp32-p4-nano.cc to support display initialization commands, enhancing the board's functionality.
* chore: Update build workflow and board configuration files
- Changed the build job name in the GitHub Actions workflow to use the full name of the matrix variant for better clarity.
- Refactored CMakeLists.txt to improve the formatting of the BOARD_SOURCES file globbing.
- Added manufacturer information to the config.json files for the eda-robot-pro, eda-super-bear, and eda-tv-pro boards to ensure consistency in configuration.
* chore: Update esp-wifi-connect version in idf_component.yml and refactor event callback in WifiBoard class
- Bumped esp-wifi-connect version from ~3.0.2 to ~3.1.1 in idf_component.yml.
- Refactored the event callback in WifiBoard to use a string parameter for SSID data, improving clarity and consistency in network event handling.
* refactor: Change logging level for OpusHead and OpusTags detection in OggDemuxer
- Updated log statements from ESP_LOGI to ESP_LOGD for OpusHead and OpusTags found events, improving log verbosity control.
* add: Moji 2 has built-in ESP32-C5 dual band Wi-Fi
* feat: PowerSaveTimer 160 >> 240
* fix(audio): add frame buffering to NoAudioProcessor
On chips where CONFIG_USE_AUDIO_PROCESSOR is unavailable (e.g., ESP32-C5,
which is excluded by Kconfig despite having PSRAM), NoAudioProcessor is used.
The original implementation passed through 160-sample chunks directly, but
the Opus encoder expects 960 samples (60ms @ 16kHz), resulting in frame size
mismatch errors, high CPU usage, and watchdog resets.
Added output_buffer_ to accumulate audio samples until reaching
frame_samples_ (960) before outputting, matching AfeAudioProcessor behavior.
other:
Turn off sleep mode (PowerSaveTimer)
- Adjusted the logic to check for low battery popup visibility only during clock tick events, ensuring it does not trigger during initialization when battery level is not ready.
- Implemented a scheduled task to play the low battery sound, improving responsiveness and avoiding immediate playback during UI updates.
* Add EDA Education Board Configuration
- Add eda-tv-pro Board Configuration
- Add eda-robot-pro Board Configuration
- Add eda-super-bear Board Configuration
* docs(oscillator): Add file headers with copyright and license information
- Add comprehensive file headers to oscillator.cc and oscillator.h in eda-robot-pro board
- Add comprehensive file headers to oscillator.cc and oscillator.h in eda-super-bear board
- Include original author attribution (Juan Gonzalez-Gomez/Obijuan) and ESP32 port credit (txp666)
- Include GPL license notice and file descriptions for clarity and compliance
* fix: Move the eda* boards to the lceda-course-examples folder.
- Move eda-robot-pro, eda-super-bear, and eda-tv-pro boards to lceda-course-examples subdirectory
- Update CMakeLists.txt to set MANUFACTURER variable for each board configuration
* chore: Update component versions and enhance UI setup across multiple boards
- Bumped uart-eth-modem version from ~0.3.2 to ~0.3.3 in idf_component.yml.
- Added SetupUI method to various display classes to ensure proper UI initialization before usage.
- Improved error handling in display classes to prevent issues when UI is not set up.
- Ensured UI customization is performed in SetupUI rather than constructors for better reliability.
* remove pm config code
* refactor: Remove hardcoded loop delay for GIF playback in LcdDisplay class
* chore: Update esp-ml307 and uart-eth-modem component versions in idf_component.yml
- Bump esp-ml307 version from ~3.6.3 to ~3.6.4
- Update uart-eth-modem version from ~0.3.1 to ~0.3.2
* feat: Add PrintPmLocks method to SystemInfo class
- Introduced PrintPmLocks method to display power management locks using esp_pm_dump_locks.
- Updated system_info.h to declare the new method.
* refactor: Streamline audio codec initialization and enablement
- Removed redundant channel enable checks from AudioCodec::Start.
- Added channel enablement in CreateDuplexChannels for various audio codecs.
- Implemented EnableInput and EnableOutput methods in NoAudioCodec for better control over input/output states.
* refactor: Delay audio success sound playback until after activation completion
- Moved the success sound playback to a scheduled task to ensure it occurs after the activation process is complete.
- This change improves the responsiveness of the application during activation events.
* refactor: Update camera integration from EspVideo to Esp32Camera
- Replaced EspVideo with Esp32Camera for improved camera configuration and initialization.
- Streamlined camera setup by utilizing a new configuration structure for better clarity and maintainability.
- Updated README.md to remove outdated camera sensor configuration instructions.
* refactor: Update audio demuxing process in AudioService
- Replaced the existing demuxer instance with a local unique pointer in the PlaySound method for better memory management.
- Moved the OnDemuxerFinished callback setup into the PlaySound method to ensure it is correctly associated with the new demuxer instance.
- Removed the member variable demuxer_ from AudioService to streamline the class structure.
content: Update the README.md
ci: Adapt to the lower directory
ci: the lower directory support
ci: board type errors fixed.
ci: board_type errors fixed to board_leaf.
fix: A naming error problem
fix: jd9365 member error
fix: Update product naming
feat: Add manufacturer parameters to the firmware name for 'release'
fix: Verification logic
fix: Verification logic
* Enhance audio processing and wake word detection
- Set task priority in Application::Run to improve responsiveness.
- Log detected wake words with their state in HandleWakeWordDetectedEvent.
- Streamline audio feeding in AudioService to handle both wake word and audio processor events.
- Implement input buffering in AfeAudioProcessor, AfeWakeWord, CustomWakeWord, and EspWakeWord to manage audio data more efficiently.
- Clear input buffers on stop to prevent residual data issues.
* Refactor audio processing to enhance thread safety and state management
- Implement early return checks in Feed methods of AfeAudioProcessor, AfeWakeWord, CustomWakeWord, and EspWakeWord to prevent processing when not running.
- Introduce std::atomic for running state in CustomWakeWord and EspWakeWord to ensure thread-safe access.
- Consolidate input buffer management with mutex locks to avoid race conditions during Stop and Feed operations.
* Refactor listening mode handling and wake word detection configuration
- Replace direct mode setting logic with a new GetDefaultListeningMode method for improved clarity and maintainability.
- Update HandleToggleChatEvent, HandleWakeWordDetectedEvent, and ContinueWakeWordInvoke to utilize the new method for determining listening mode.
- Introduce Kconfig option WAKE_WORD_DETECTION_IN_LISTENING to enable or disable wake word detection during listening mode, enhancing configurability.
* Fix: uart-uhci compiling errors
* Enhance Esp32Camera functionality by adding optional byte swapping for RGB565 format. Introduce SetSwapBytes method to enable/disable byte order swapping, and update Capture method to utilize an encode buffer for improved memory management and performance during image processing.
* Update esp-ml307 dependency version to ~3.6.0 in idf_component.yml
* Update .gitignore to include 'dist/' directory, add ml307 and dual_network_board source files to CMakeLists.txt, and update esp-ml307 dependency version to ~3.6.2 in idf_component.yml. Refactor CompactWifiBoard and CompactWifiBoardLCD classes to inherit from WifiBoard instead of DualNetworkBoard, simplifying network handling logic.
- Introduced ContinueOpenAudioChannel and ContinueWakeWordInvoke methods to streamline audio channel management and wake word processing.
- Updated HandleToggleChatEvent and HandleWakeWordDetectedEvent to utilize scheduling for state changes, improving UI responsiveness.
- Simplified logic for setting listening modes based on audio channel state, enhancing code clarity and maintainability.
* Enhance memory management in asset download and OTA processes by replacing static buffer allocations with dynamic memory allocation using heap capabilities. Update SPIRAM configuration values for improved memory usage. Add logging for error handling in buffer allocation failures. Introduce a new parameter in CloseAudioChannel to control goodbye message sending in MQTT and WebSocket protocols.
* Update component versions in idf_component.yml and refactor GIF decoder functions for improved performance. Bump versions for audio effects, audio codec, LED strip, and other dependencies. Change GIF read and seek functions to inline for optimization.
* Update language files to include new phrases for flight mode and connection status across multiple locales. Added translations for "FLIGHT_MODE_ON", "FLIGHT_MODE_OFF", "CONNECTION_SUCCESSFUL", and "MODEM_INIT_ERROR" in various languages, enhancing user experience and localization support.
* fix wechat display
* otto v1.4.0 MCP
1.使用MCP协议控制机器人
2.gif继承lcdDisplay,避免修改lcdDisplay
* otto v1.4.1 gif as components
gif as components
* electronBot v1.1.0 mcp
1.增加electronBot支持
2.mcp协议
3.gif 作为组件
4.display子类
* 规范代码
1.规范代码
2.修复切换主题死机bug
* fix(ota): 修复 ottoRobot和electronBot OTA 升级崩溃问题 bug
* 1.增加robot舵机初始位置校准
2.fix(mcp_sever) 超出范围异常捕获类型 bug
* refactor: Update Electron and Otto emoji display implementations
- Removed GIF selection from Kconfig for Electron and Otto boards.
- Updated Electron and Otto bot versions to 2.0.4 in their respective config files.
- Refactored emoji display classes to utilize EmojiCollection for managing emojis.
- Enhanced chat label setup and status display functionality in both classes.
- Cleaned up unused code and improved initialization logging for emoji displays.
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* refactor: Update Otto emoji display configurations and functionalities
- Changed chat label text mode to circular scrolling for both Otto and Electron emoji displays.
- Bumped Otto robot version to 2.0.5 in the configuration file.
- Added new actions for Otto robot including Sit, WhirlwindLeg, Fitness, Greeting, Shy, RadioCalisthenics, MagicCircle, and Showcase.
- Enhanced servo sequence handling and added support for executing custom servo sequences.
- Improved logging and error handling for servo sequence execution.
* refactor: Update chat label long mode for Electron and Otto emoji displays
- Changed chat label text mode from wrap to circular scrolling for both Electron and Otto emoji displays.
- Improved consistency in chat label setup across both implementations.
* Update Otto robot README with new actions and parameters
* Update Otto controller parameters for oscillation settings
- Changed default oscillation period from 500ms to 300ms.
- Increased default steps from 5.0 to 8.0.
- Updated default amplitude from 20 degrees to 0 degrees.
- Enhanced documentation with new examples for oscillation modes and sequences.
* Fix default amplitude initialization in Otto controller to use a single zero instead of two digits.
* chore: update txp666/otto-emoji-gif-component version to 1.0.3 in idf_component.yml
* Refactor Otto controller
- Consolidated movement actions into a unified tool for the Otto robot, allowing for a single action command with various parameters.
- Removed individual movement tools (walk, turn, jump, etc.) and replaced them with a more flexible action system.
* Enhance Otto robot functionality by adding WebSocket control server and IP address retrieval feature. Updated config to support WebSocket, and revised README to include new control options and usage examples.
* Add camera support for Otto Robot board
- Introduced configuration option to enable the Otto Robot camera in Kconfig.
- Updated config.h to define camera-related GPIO pins and settings.
- Modified config.json to include camera configuration.
- Enhanced otto_robot.cc to initialize I2C and camera components when the camera is enabled.
- Adjusted power_manager.h to manage battery updates during camera operations.
- Removed unused SetupChatLabel method from OttoEmojiDisplay class.
* Refactor Otto Robot configuration and initialization
- Removed the camera configuration option from Kconfig and related code.
- Introduced a new HardwareConfig struct to encapsulate hardware pin definitions and settings.
- Updated config.h to define camera and non-camera configurations using the new struct.
- Refactored otto_controller.cc and otto_robot.cc to utilize the HardwareConfig struct for initialization.
- Enhanced camera detection and initialization logic based on hardware version.
- Improved audio codec initialization based on configuration settings.
* Refactor emoji initialization for Electron and Otto boards to use Assets system
- Removed direct emoji initialization from `InitializeElectronEmojis` and `InitializeOttoEmojis` methods, delegating the responsibility to the Assets system.
- Updated `CMakeLists.txt` to set `DEFAULT_EMOJI_COLLECTION` to `otto-gif` for both boards.
- Enhanced `build_default_assets.py` to support alias mapping for Otto GIF emojis.
- Updated `idf_component.yml` to bump `otto-emoji-gif-component` version to `^1.0.5` for improved functionality.
* Initial plan
* Fix LVGL object deletion issue in SetChatMessage
- Refresh child_count after deleting first_child to avoid using stale count
- Add lv_obj_is_valid() checks before accessing deleted objects
- Prevent duplicate deletion by refreshing child_count before system message deletion
- Ensures last_child validation before scrolling to it
Co-authored-by: 78 <4488133+78@users.noreply.github.com>
* Address code review feedback on LVGL object deletion fix
- Move last_child retrieval after deletion to avoid stale pointer
- Add child_count > 0 check before scrolling to prevent edge case issues
- Add lv_obj_is_valid() check for last_bubble for consistency
Co-authored-by: 78 <4488133+78@users.noreply.github.com>
* Final review: LVGL object deletion fix complete
Co-authored-by: 78 <4488133+78@users.noreply.github.com>
* Remove std::thread wrappers in display callbacks to fix thread safety
Remove detached threads that were calling SetChatMessage from download/upgrade
progress callbacks. The DisplayLockGuard mutex already provides thread safety,
so we can call display methods directly from the callback thread without
spawning additional threads, which were causing race conditions.
Co-authored-by: 78 <4488133+78@users.noreply.github.com>
* Use Application::Schedule() for display updates in callbacks
Instead of calling SetChatMessage directly from download/upgrade progress
callbacks, queue the updates using Application::Schedule(). This prevents
blocking the network receive task and avoids potential UART FIFO overflow
issues with 4G modems. The scheduled callbacks execute in the main task
thread, maintaining proper thread safety.
Co-authored-by: 78 <4488133+78@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: 78 <4488133+78@users.noreply.github.com>
* Add support for RNDIS board and enhance camera initialization
- Included rndis_board.cc in the build for ESP32S3 and ESP32P4 targets.
- Updated camera initialization logic in esp32s3_korvo2_v3_board.cc and esp32s3_korvo2_v3_board.cc to use a more structured camera_config_t setup.
- Improved code readability by refining comments and formatting in the camera initialization functions.
* Remove outdated camera configuration options from esp32s3-korvo2-v3-rndis config.json to streamline setup and improve clarity.
* Update IDF version in build configuration and component dependencies to v5.5.2 for improved compatibility.
* update discord links
---------
Co-authored-by: Xiaoxia <terrence.huang@tenclass.com>
* Update project version to 2.2.1 and refactor camera component handling
- Incremented project version from 2.2.0 to 2.2.1 in CMakeLists.txt.
- Removed legacy esp32_camera component and replaced it with esp_video for ESP32-S3 and ESP32-P4 boards.
- Updated board implementations to utilize the new esp_video component, ensuring compatibility and improved functionality.
- Cleaned up Kconfig options related to camera selection, streamlining the configuration process.
- Enhanced camera initialization logic across various board files to support the new component structure.
* Refactor camera handling in AtomS3R CAM/M12 EchoBase board
- Replaced the legacy EspVideo component with the new Esp32Camera class for improved camera functionality.
- Updated camera initialization logic to utilize a more structured configuration approach, enhancing clarity and maintainability.
- Removed outdated comments and code related to the previous camera implementation in the README file.
* Update camera configuration for atoms3r-cam-m12-echo-base
- Removed outdated camera configuration options from config.json to streamline the setup.
- Retained essential partition table configuration for improved clarity.
* Enhance Esp32Camera functionality and memory management
- Added esp_timer.h for improved timing functionality.
- Streamlined camera initialization by removing redundant frame buffer setup and logging.
- Improved memory allocation for JPEG encoding and added error handling for unsupported pixel formats.
- Updated comments for clarity and consistency, ensuring better understanding of the code flow.
Introduces a new esp32s3_camera implementation for ESP32-S3 boards using the esp_camera component, with conditional compilation and Kconfig options to select between esp_camera and esp_video. Updates board initialization code and config files to use the new camera class where appropriate, and adjusts build system and dependencies to support both camera components on ESP32-S3 and ESP32-P4 targets.
* Refactor application error handling and improve network task logic
- Updated error handling for modem initialization failure in Application::Initialize().
- Added new error message for modem initialization in English and Chinese language files.
- Simplified lambda captures in NetworkTask to avoid unnecessary references.
- Set main task priority in Application::Run() for better performance.
* Add support for Bread Compact NT26 board
- Introduced new board configuration for Bread Compact NT26 in CMakeLists.txt and Kconfig.
- Added board-specific implementation in compact_nt26_board.cc and nt26_board.cc.
- Created configuration files for NT26, including config.h and config.json.
- Updated dependencies in idf_component.yml to include uart-eth-modem.
- Translated error messages in config.h for OLED display type selection to English.
- Enhanced display and button initialization logic for NT26 board.
* Update project version and improve build configuration
- Updated project version from 2.1.0 to 2.2.0 in CMakeLists.txt.
- Enabled minimal build configuration to include only essential components.
- Updated README files to replace QQ group links with Discord links for community engagement.
* Update Bread Compact NT26 board configuration name in config.json
* fix compile errors
* Update uart-eth-modem dependency format in idf_component.yml
* fix esp32 compiling errors
* Update CMakeLists.txt to change component dependency from REQUIRES to PRIV_REQUIRES for esp_pm, esp_psram, and esp_driver_gpio
* Refactor CMakeLists.txt to explicitly list board common source files and update include directories for better clarity and organization.
* Add esp_driver_ppa as a dependency in CMakeLists.txt
* content: New support for third-party boards: waveshare ESP32-C6-Touch-AMOLED-1.8
* content: Change the TAG of ESP32-C6-Touch-AMOLED-1.8
* content: Change the config.json of ESP32-C6-Touch-AMOLED-1.8
Introduces PowerManager and integrates it into the ESP32S3-Korvo2-V3 board class to monitor battery level, charging status, and manage power save mode. Adds power_manager.h with battery ADC reading, calibration, and event callbacks. Updates board initialization to support power management and power save timer functionality.
Added camera-related sdkconfig options to esp32s3-korvo2-v3 and kevin-sp-v4-dev config.json files. Updated esp32s3_korvo2_v3_board.cc to implement button actions for WiFi config and chat state toggling. Refactored kevin-sp-v4_board.cc to unify I2C bus usage for codec and camera, and adjusted SCCB initialization logic.
* Enhance audio feedback mechanism by introducing a flag to play a popup sound after transitioning to listening mode. Update Schedule method to accept rvalue references for callbacks. Bump esp-ml307 component version to 3.5.3. Adjust signal strength thresholds in Ml307Board for better accuracy.
* Update esp-wifi-connect component version to 3.0.2 in idf_component.yml
* Adjust Wi-Fi signal strength thresholds in WifiBoard for improved accuracy in network state icon representation.
* Upgrade component version
* update fonts component version
* Handle OTA error code
* Update project version to 2.1.0 and add device state machine implementation
- Upgrade esp-wifi-connect to 3.0.0, allowing reconfiguring wifi without rebooting
- Introduce device state machine with state change notification in new files
- Remove obsolete device state event files
- Update application logic to utilize new state machine
- Minor adjustments in various board implementations for state handling
* fix compile errors
* Refactor power saving mode implementation to use PowerSaveLevel enumeration
- Updated Application class to replace SetPowerSaveMode with SetPowerSaveLevel, allowing for LOW_POWER and PERFORMANCE settings.
- Modified various board implementations to align with the new power save level structure.
- Ensured consistent handling of power save levels across different board files, enhancing code maintainability and clarity.
* Refactor power save level checks across multiple board implementations
- Updated the condition for power save level checks in various board files to ensure that the power save timer only wakes up when the level is not set to LOW_POWER.
- Improved consistency in handling power save levels, enhancing code clarity and maintainability.
* Refactor EnterWifiConfigMode calls in board implementations
- Updated calls to EnterWifiConfigMode to use the appropriate instance reference (self or board) across multiple board files.
- Improved code consistency and clarity in handling device state during WiFi configuration mode entry.
* Add cellular modem event handling and improve network status updates
- Introduced new network events for cellular modem operations, including detecting, registration errors, and timeouts.
- Enhanced the Application class to handle different network states and update the display status accordingly.
- Refactored Ml307Board to implement a callback mechanism for network events, improving modularity and responsiveness.
- Updated dual_network_board and board headers to support new network event callbacks, ensuring consistent handling across board implementations.
* update esp-wifi-connect version
* Update WiFi configuration tool messages across multiple board implementations to clarify user actions
* otto v1.4.0 MCP
1.使用MCP协议控制机器人
2.gif继承lcdDisplay,避免修改lcdDisplay
* otto v1.4.1 gif as components
gif as components
* electronBot v1.1.0 mcp
1.增加electronBot支持
2.mcp协议
3.gif 作为组件
4.display子类
* 规范代码
1.规范代码
2.修复切换主题死机bug
* fix(ota): 修复 ottoRobot和electronBot OTA 升级崩溃问题 bug
* 1.增加robot舵机初始位置校准
2.fix(mcp_sever) 超出范围异常捕获类型 bug
* refactor: Update Electron and Otto emoji display implementations
- Removed GIF selection from Kconfig for Electron and Otto boards.
- Updated Electron and Otto bot versions to 2.0.4 in their respective config files.
- Refactored emoji display classes to utilize EmojiCollection for managing emojis.
- Enhanced chat label setup and status display functionality in both classes.
- Cleaned up unused code and improved initialization logging for emoji displays.
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* refactor: Update Otto emoji display configurations and functionalities
- Changed chat label text mode to circular scrolling for both Otto and Electron emoji displays.
- Bumped Otto robot version to 2.0.5 in the configuration file.
- Added new actions for Otto robot including Sit, WhirlwindLeg, Fitness, Greeting, Shy, RadioCalisthenics, MagicCircle, and Showcase.
- Enhanced servo sequence handling and added support for executing custom servo sequences.
- Improved logging and error handling for servo sequence execution.
* refactor: Update chat label long mode for Electron and Otto emoji displays
- Changed chat label text mode from wrap to circular scrolling for both Electron and Otto emoji displays.
- Improved consistency in chat label setup across both implementations.
* Update Otto robot README with new actions and parameters
* Update Otto controller parameters for oscillation settings
- Changed default oscillation period from 500ms to 300ms.
- Increased default steps from 5.0 to 8.0.
- Updated default amplitude from 20 degrees to 0 degrees.
- Enhanced documentation with new examples for oscillation modes and sequences.
* Fix default amplitude initialization in Otto controller to use a single zero instead of two digits.
* chore: update txp666/otto-emoji-gif-component version to 1.0.3 in idf_component.yml
* Refactor Otto controller
- Consolidated movement actions into a unified tool for the Otto robot, allowing for a single action command with various parameters.
- Removed individual movement tools (walk, turn, jump, etc.) and replaced them with a more flexible action system.
* Enhance Otto robot functionality by adding WebSocket control server and IP address retrieval feature. Updated config to support WebSocket, and revised README to include new control options and usage examples.
* Add camera support for Otto Robot board
- Introduced configuration option to enable the Otto Robot camera in Kconfig.
- Updated config.h to define camera-related GPIO pins and settings.
- Modified config.json to include camera configuration.
- Enhanced otto_robot.cc to initialize I2C and camera components when the camera is enabled.
- Adjusted power_manager.h to manage battery updates during camera operations.
- Removed unused SetupChatLabel method from OttoEmojiDisplay class.
* Refactor Otto Robot configuration and initialization
- Removed the camera configuration option from Kconfig and related code.
- Introduced a new HardwareConfig struct to encapsulate hardware pin definitions and settings.
- Updated config.h to define camera and non-camera configurations using the new struct.
- Refactored otto_controller.cc and otto_robot.cc to utilize the HardwareConfig struct for initialization.
- Enhanced camera detection and initialization logic based on hardware version.
- Improved audio codec initialization based on configuration settings.
* otto v1.4.0 MCP
1.使用MCP协议控制机器人
2.gif继承lcdDisplay,避免修改lcdDisplay
* otto v1.4.1 gif as components
gif as components
* electronBot v1.1.0 mcp
1.增加electronBot支持
2.mcp协议
3.gif 作为组件
4.display子类
* 规范代码
1.规范代码
2.修复切换主题死机bug
* fix(ota): 修复 ottoRobot和electronBot OTA 升级崩溃问题 bug
* 1.增加robot舵机初始位置校准
2.fix(mcp_sever) 超出范围异常捕获类型 bug
* refactor: Update Electron and Otto emoji display implementations
- Removed GIF selection from Kconfig for Electron and Otto boards.
- Updated Electron and Otto bot versions to 2.0.4 in their respective config files.
- Refactored emoji display classes to utilize EmojiCollection for managing emojis.
- Enhanced chat label setup and status display functionality in both classes.
- Cleaned up unused code and improved initialization logging for emoji displays.
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* refactor: Update Otto emoji display configurations and functionalities
- Changed chat label text mode to circular scrolling for both Otto and Electron emoji displays.
- Bumped Otto robot version to 2.0.5 in the configuration file.
- Added new actions for Otto robot including Sit, WhirlwindLeg, Fitness, Greeting, Shy, RadioCalisthenics, MagicCircle, and Showcase.
- Enhanced servo sequence handling and added support for executing custom servo sequences.
- Improved logging and error handling for servo sequence execution.
* refactor: Update chat label long mode for Electron and Otto emoji displays
- Changed chat label text mode from wrap to circular scrolling for both Electron and Otto emoji displays.
- Improved consistency in chat label setup across both implementations.
* Update Otto robot README with new actions and parameters
* Update Otto controller parameters for oscillation settings
- Changed default oscillation period from 500ms to 300ms.
- Increased default steps from 5.0 to 8.0.
- Updated default amplitude from 20 degrees to 0 degrees.
- Enhanced documentation with new examples for oscillation modes and sequences.
* Fix default amplitude initialization in Otto controller to use a single zero instead of two digits.
* chore: update txp666/otto-emoji-gif-component version to 1.0.3 in idf_component.yml
* Refactor Otto controller
- Consolidated movement actions into a unified tool for the Otto robot, allowing for a single action command with various parameters.
- Removed individual movement tools (walk, turn, jump, etc.) and replaced them with a more flexible action system.
* Enhance Otto robot functionality by adding WebSocket control server and IP address retrieval feature. Updated config to support WebSocket, and revised README to include new control options and usage examples.
* Add camera support for Otto Robot board
- Introduced configuration option to enable the Otto Robot camera in Kconfig.
- Updated config.h to define camera-related GPIO pins and settings.
- Modified config.json to include camera configuration.
- Enhanced otto_robot.cc to initialize I2C and camera components when the camera is enabled.
- Adjusted power_manager.h to manage battery updates during camera operations.
- Removed unused SetupChatLabel method from OttoEmojiDisplay class.
* Added new board for hu-087 smart AI watch
* Added a README with link to prouct on Aliexpress
* Trying to control chatbot mode instead of listening state
* Removed the esp32-s3 prefix for the board
* Changed board name to match folder naming
* Fixed formating and long URL
* Fixed typo in README.md
* feat: add keepalive check for Himax client and handle restart
* fix: adjust layout and positioning for top and bottom bars
* fix: fix other param restoring default when modifying one param .
* Upgrade component version
* update fonts component version
* change lcd display layout from grids to layers
* Update English README as default
* Handle OTA error code
* otto v1.4.0 MCP
1.使用MCP协议控制机器人
2.gif继承lcdDisplay,避免修改lcdDisplay
* otto v1.4.1 gif as components
gif as components
* electronBot v1.1.0 mcp
1.增加electronBot支持
2.mcp协议
3.gif 作为组件
4.display子类
* 规范代码
1.规范代码
2.修复切换主题死机bug
* fix(ota): 修复 ottoRobot和electronBot OTA 升级崩溃问题 bug
* 1.增加robot舵机初始位置校准
2.fix(mcp_sever) 超出范围异常捕获类型 bug
* refactor: Update Electron and Otto emoji display implementations
- Removed GIF selection from Kconfig for Electron and Otto boards.
- Updated Electron and Otto bot versions to 2.0.4 in their respective config files.
- Refactored emoji display classes to utilize EmojiCollection for managing emojis.
- Enhanced chat label setup and status display functionality in both classes.
- Cleaned up unused code and improved initialization logging for emoji displays.
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* refactor: Update Otto emoji display configurations and functionalities
- Changed chat label text mode to circular scrolling for both Otto and Electron emoji displays.
- Bumped Otto robot version to 2.0.5 in the configuration file.
- Added new actions for Otto robot including Sit, WhirlwindLeg, Fitness, Greeting, Shy, RadioCalisthenics, MagicCircle, and Showcase.
- Enhanced servo sequence handling and added support for executing custom servo sequences.
- Improved logging and error handling for servo sequence execution.
* refactor: Update chat label long mode for Electron and Otto emoji displays
- Changed chat label text mode from wrap to circular scrolling for both Electron and Otto emoji displays.
- Improved consistency in chat label setup across both implementations.
* Update Otto robot README with new actions and parameters
* Update Otto controller parameters for oscillation settings
- Changed default oscillation period from 500ms to 300ms.
- Increased default steps from 5.0 to 8.0.
- Updated default amplitude from 20 degrees to 0 degrees.
- Enhanced documentation with new examples for oscillation modes and sequences.
* Fix default amplitude initialization in Otto controller to use a single zero instead of two digits.
* chore: update txp666/otto-emoji-gif-component version to 1.0.3 in idf_component.yml
* Refactor Otto controller
- Consolidated movement actions into a unified tool for the Otto robot, allowing for a single action command with various parameters.
- Removed individual movement tools (walk, turn, jump, etc.) and replaced them with a more flexible action system.
* Enhance Otto robot functionality by adding WebSocket control server and IP address retrieval feature. Updated config to support WebSocket, and revised README to include new control options and usage examples.
* Add support for the Waveshare ESP32-C6-Touch-AMOLED-2.06 third-party board
* Fix the known issues
* Fix the name
* Update screen brightness
* readme content error correction
* Update the default wake-up word configuration for the ESP32-P4 chip
* Refactor DSI bus configuration for LCD panel
* Update MIPI DSI bus configuration for LCD
* Update MIPI DSI bus configuration structure
* Cancel redundant configuration
* Cancel redundant configuration
* feat: add support for ESP32-P4-Function-EV-Board with configuration and display handling
* detect wake word model from index.json
* update wait time before entering wifi configure mode
* feat: Enhance ESP32-P4 Function EV Board support with LCD and touch initialization
* feat: Update ESP32-P4 Function EV Board configuration for improved touch and SD card support
* feat: add touch I2C configuration and improve initialization structure
* Remove ESP hosted configuration from defaults
Removed ESP hosted configuration options.
* chore: update documentation for improved clarity
* refactor: remove obsolete files for ESP32-P4-Function-EV-Board and add updated configurations
* refactor: reintroduce ESP32-P4-Function-EV-Board implementation with updated configurations
* refactor: restore esp32_p4_function_ev_board dependency with updated version
* Remove ESP-P4-Function-EV-Board configuration and update README with detailed features; add SD card, camera, and font initialization in esp-p4-function-ev-board.cc
* Update main/boards/esp-p4-function-ev-board/esp-p4-function-ev-board.cc
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update main/boards/esp-p4-function-ev-board/esp-p4-function-ev-board.cc
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Enhance camera initialization in ESP-P4-Function-EV-Board; add fallback configuration for direct camera setup and update README for clarity
* Update clock source for MIPI DSI configuration in ESP-P4-Function-EV-Board
---------
Co-authored-by: n2flowjs-bot <n2flowjs@gmail.com>
Co-authored-by: Terrence <terrence@tenclass.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* otto v1.4.0 MCP
1.使用MCP协议控制机器人
2.gif继承lcdDisplay,避免修改lcdDisplay
* otto v1.4.1 gif as components
gif as components
* electronBot v1.1.0 mcp
1.增加electronBot支持
2.mcp协议
3.gif 作为组件
4.display子类
* 规范代码
1.规范代码
2.修复切换主题死机bug
* fix(ota): 修复 ottoRobot和electronBot OTA 升级崩溃问题 bug
* 1.增加robot舵机初始位置校准
2.fix(mcp_sever) 超出范围异常捕获类型 bug
* refactor: Update Electron and Otto emoji display implementations
- Removed GIF selection from Kconfig for Electron and Otto boards.
- Updated Electron and Otto bot versions to 2.0.4 in their respective config files.
- Refactored emoji display classes to utilize EmojiCollection for managing emojis.
- Enhanced chat label setup and status display functionality in both classes.
- Cleaned up unused code and improved initialization logging for emoji displays.
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* refactor: Update Otto emoji display configurations and functionalities
- Changed chat label text mode to circular scrolling for both Otto and Electron emoji displays.
- Bumped Otto robot version to 2.0.5 in the configuration file.
- Added new actions for Otto robot including Sit, WhirlwindLeg, Fitness, Greeting, Shy, RadioCalisthenics, MagicCircle, and Showcase.
- Enhanced servo sequence handling and added support for executing custom servo sequences.
- Improved logging and error handling for servo sequence execution.
* refactor: Update chat label long mode for Electron and Otto emoji displays
- Changed chat label text mode from wrap to circular scrolling for both Electron and Otto emoji displays.
- Improved consistency in chat label setup across both implementations.
* Update Otto robot README with new actions and parameters
* Update Otto controller parameters for oscillation settings
- Changed default oscillation period from 500ms to 300ms.
- Increased default steps from 5.0 to 8.0.
- Updated default amplitude from 20 degrees to 0 degrees.
- Enhanced documentation with new examples for oscillation modes and sequences.
* Fix default amplitude initialization in Otto controller to use a single zero instead of two digits.
* chore: update txp666/otto-emoji-gif-component version to 1.0.3 in idf_component.yml
* Refactor Otto controller
- Consolidated movement actions into a unified tool for the Otto robot, allowing for a single action command with various parameters.
- Removed individual movement tools (walk, turn, jump, etc.) and replaced them with a more flexible action system.
* otto v1.4.0 MCP
1.使用MCP协议控制机器人
2.gif继承lcdDisplay,避免修改lcdDisplay
* otto v1.4.1 gif as components
gif as components
* electronBot v1.1.0 mcp
1.增加electronBot支持
2.mcp协议
3.gif 作为组件
4.display子类
* 规范代码
1.规范代码
2.修复切换主题死机bug
* fix(ota): 修复 ottoRobot和electronBot OTA 升级崩溃问题 bug
* 1.增加robot舵机初始位置校准
2.fix(mcp_sever) 超出范围异常捕获类型 bug
* refactor: Update Electron and Otto emoji display implementations
- Removed GIF selection from Kconfig for Electron and Otto boards.
- Updated Electron and Otto bot versions to 2.0.4 in their respective config files.
- Refactored emoji display classes to utilize EmojiCollection for managing emojis.
- Enhanced chat label setup and status display functionality in both classes.
- Cleaned up unused code and improved initialization logging for emoji displays.
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* refactor: Update Otto emoji display configurations and functionalities
- Changed chat label text mode to circular scrolling for both Otto and Electron emoji displays.
- Bumped Otto robot version to 2.0.5 in the configuration file.
- Added new actions for Otto robot including Sit, WhirlwindLeg, Fitness, Greeting, Shy, RadioCalisthenics, MagicCircle, and Showcase.
- Enhanced servo sequence handling and added support for executing custom servo sequences.
- Improved logging and error handling for servo sequence execution.
* refactor: Update chat label long mode for Electron and Otto emoji displays
- Changed chat label text mode from wrap to circular scrolling for both Electron and Otto emoji displays.
- Improved consistency in chat label setup across both implementations.
* Update Otto robot README with new actions and parameters
* Update Otto controller parameters for oscillation settings
- Changed default oscillation period from 500ms to 300ms.
- Increased default steps from 5.0 to 8.0.
- Updated default amplitude from 20 degrees to 0 degrees.
- Enhanced documentation with new examples for oscillation modes and sequences.
* Fix default amplitude initialization in Otto controller to use a single zero instead of two digits.
* chore: update txp666/otto-emoji-gif-component version to 1.0.3 in idf_component.yml
* otto v1.4.0 MCP
1.使用MCP协议控制机器人
2.gif继承lcdDisplay,避免修改lcdDisplay
* otto v1.4.1 gif as components
gif as components
* electronBot v1.1.0 mcp
1.增加electronBot支持
2.mcp协议
3.gif 作为组件
4.display子类
* 规范代码
1.规范代码
2.修复切换主题死机bug
* fix(ota): 修复 ottoRobot和electronBot OTA 升级崩溃问题 bug
* 1.增加robot舵机初始位置校准
2.fix(mcp_sever) 超出范围异常捕获类型 bug
* refactor: Update Electron and Otto emoji display implementations
- Removed GIF selection from Kconfig for Electron and Otto boards.
- Updated Electron and Otto bot versions to 2.0.4 in their respective config files.
- Refactored emoji display classes to utilize EmojiCollection for managing emojis.
- Enhanced chat label setup and status display functionality in both classes.
- Cleaned up unused code and improved initialization logging for emoji displays.
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* refactor: Update Otto emoji display configurations and functionalities
- Changed chat label text mode to circular scrolling for both Otto and Electron emoji displays.
- Bumped Otto robot version to 2.0.5 in the configuration file.
- Added new actions for Otto robot including Sit, WhirlwindLeg, Fitness, Greeting, Shy, RadioCalisthenics, MagicCircle, and Showcase.
- Enhanced servo sequence handling and added support for executing custom servo sequences.
- Improved logging and error handling for servo sequence execution.
* refactor: Update chat label long mode for Electron and Otto emoji displays
- Changed chat label text mode from wrap to circular scrolling for both Electron and Otto emoji displays.
- Improved consistency in chat label setup across both implementations.
* Update Otto robot README with new actions and parameters
* Update Otto controller parameters for oscillation settings
- Changed default oscillation period from 500ms to 300ms.
- Increased default steps from 5.0 to 8.0.
- Updated default amplitude from 20 degrees to 0 degrees.
- Enhanced documentation with new examples for oscillation modes and sequences.
* Fix default amplitude initialization in Otto controller to use a single zero instead of two digits.
* otto v1.4.0 MCP
1.使用MCP协议控制机器人
2.gif继承lcdDisplay,避免修改lcdDisplay
* otto v1.4.1 gif as components
gif as components
* electronBot v1.1.0 mcp
1.增加electronBot支持
2.mcp协议
3.gif 作为组件
4.display子类
* 规范代码
1.规范代码
2.修复切换主题死机bug
* fix(ota): 修复 ottoRobot和electronBot OTA 升级崩溃问题 bug
* 1.增加robot舵机初始位置校准
2.fix(mcp_sever) 超出范围异常捕获类型 bug
* refactor: Update Electron and Otto emoji display implementations
- Removed GIF selection from Kconfig for Electron and Otto boards.
- Updated Electron and Otto bot versions to 2.0.4 in their respective config files.
- Refactored emoji display classes to utilize EmojiCollection for managing emojis.
- Enhanced chat label setup and status display functionality in both classes.
- Cleaned up unused code and improved initialization logging for emoji displays.
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* refactor: Update Otto emoji display configurations and functionalities
- Changed chat label text mode to circular scrolling for both Otto and Electron emoji displays.
- Bumped Otto robot version to 2.0.5 in the configuration file.
- Added new actions for Otto robot including Sit, WhirlwindLeg, Fitness, Greeting, Shy, RadioCalisthenics, MagicCircle, and Showcase.
- Enhanced servo sequence handling and added support for executing custom servo sequences.
- Improved logging and error handling for servo sequence execution.
* refactor: Update chat label long mode for Electron and Otto emoji displays
- Changed chat label text mode from wrap to circular scrolling for both Electron and Otto emoji displays.
- Improved consistency in chat label setup across both implementations.
* Update Otto robot README with new actions and parameters
* Add support for the Xorigin AiPi Lite
* aipi-lite: correction to README_en.md
* aipi-lite: use 20MHz display clock
Highger speeds might work sometimes but best not to push
it and use a sane speed.
* aipi-lite: change English name and remove Chinese TTS
* otto v1.4.0 MCP
1.使用MCP协议控制机器人
2.gif继承lcdDisplay,避免修改lcdDisplay
* otto v1.4.1 gif as components
gif as components
* electronBot v1.1.0 mcp
1.增加electronBot支持
2.mcp协议
3.gif 作为组件
4.display子类
* 规范代码
1.规范代码
2.修复切换主题死机bug
* fix(ota): 修复 ottoRobot和electronBot OTA 升级崩溃问题 bug
* 1.增加robot舵机初始位置校准
2.fix(mcp_sever) 超出范围异常捕获类型 bug
* refactor: Update Electron and Otto emoji display implementations
- Removed GIF selection from Kconfig for Electron and Otto boards.
- Updated Electron and Otto bot versions to 2.0.4 in their respective config files.
- Refactored emoji display classes to utilize EmojiCollection for managing emojis.
- Enhanced chat label setup and status display functionality in both classes.
- Cleaned up unused code and improved initialization logging for emoji displays.
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* Rename OTTO_ICON_FONT.c to otto_icon_font.c
* feat: add support for ESP32-P4-Function-EV-Board with configuration and display handling
* detect wake word model from index.json
* update wait time before entering wifi configure mode
* feat: Enhance ESP32-P4 Function EV Board support with LCD and touch initialization
* feat: Update ESP32-P4 Function EV Board configuration for improved touch and SD card support
* feat: add touch I2C configuration and improve initialization structure
* Remove ESP hosted configuration from defaults
Removed ESP hosted configuration options.
* chore: update documentation for improved clarity
* refactor: remove obsolete files for ESP32-P4-Function-EV-Board and add updated configurations
* refactor: reintroduce ESP32-P4-Function-EV-Board implementation with updated configurations
* refactor: restore esp32_p4_function_ev_board dependency with updated version
---------
Co-authored-by: n2flowjs-bot <n2flowjs@gmail.com>
Co-authored-by: Terrence <terrence@tenclass.com>
* 同步esp-video组件驱动摄像头的配置
* 修改Waveshare ESP32-S3-AUDIO-Board 摄像头配置。
* Modify the XCLK parameters of the camera, and modify the config.txt to be compatible with 2640 and 5640.
* Update the esp_video component version to 1.3.0
* Downgrade the esp_video component version. There are unfixed bugs.
* Update component version
* feat: Wake up when a person is detected
* fix: Solve the problem of no sound when using WakeWordInvoke
* fix: Solve the problem of triggering dialogue when the person has not left
* feat(vision): 优化视觉检测逻辑并增加配置接口
本次提交旨在优化视觉检测功能,使其行为更自然、更智能,并为用户提供灵活的配置选项。
主要更新包括:
1. 引入了更精细的检测状态机:
- IDLE: 空闲状态,等待检测目标。
- VALIDATING: 验证状态,在检测到目标后,持续一段时间(可配置)以确认其存在,防止误触发。
- COOLDOWN: 冷却状态,在一次成功交互后进入,避免过于频繁的打扰。
2. 新增了用于配置视觉检测的 MCP 工具:
- self.vision.get_detection_config: 获取当前的检测参数(阈值、冷却间隔、验证时长、目标类型)。
- self.vision.set_detection_config: 允许用户动态修改这些参数,以适应不同场景。
3. 性能优化:
- 增加了配置参数的内存缓存,避免了在检测循环中对 NVS 的频繁访问。
* feat: Inference using Model 4
* feat: default inference disable
* feat: version cmd change to output json
* fix: fix image display
* Fix include directives for esp_check and esp_app_desc
---------
Co-authored-by: Spencer <love4yzp@gmail.com>
Co-authored-by: Xiaoxia <terrence@tenclass.com>
* refactor: migrate camera module to esp-video library
* refactor: migrate boards to esp-video API (1/2)
* refactor: migrate boards to esp-video API (2/2)
* fix: use ESP-IDF 5.5
* refactor: migrate the JPEG encoder to `esp_new_jpeg`
* feat: add YUV422 support
* feat: improve pixelformat and device selection process
* feat: use ESP32-P4 Hardware JPEG Encoder
* add support for Wireless-Tag WTP4C5MP07S ESP32P4 + 7 inch MIPI DSI LCD display combo
* a minor update of README
* update of the device name/folder
* exclude USE_DEVICE_AEC from the options list
* Adapt for LilyGO-T-Circle-S3 device
* Adapt for LilyGO-T-Circle-S3 device
* Remove comments and modify the size of the lilygo-t-circle-s3 image
* Modify the code style and format to Google C++
* Modify the code style and format to Google C++
* Fixed bugs in the LILYGO T-Circle-S3 board and added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Merge branch 'main' of https://github.com/Llgok/xiaozhi-esp32
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Fix the color display issue for T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Update T-CameraPlus-S3_V1.2 Version Xiaozhi Example
* Resolve the issue where the camera on the T-CameraPlus-S3_V1.2 board cannot be used normally.
* Enhance microphone reception volume
* fix the issue where voice wake-up is not working
* fix the issue where voice wake-up is not working
* Add LILYGO T-Display-P4 board adaptation
* Modify the MIPILCDDISPLAY class
* Added Waveshare ESP32-P4-WIFI6-Touch-LCD-7B third party board
* Add config,json for Waveshare ESP32-P4 series boards
* Fix some errors
* update v2 partition table readme
* feat: Add user only tool
* Add image cache
* smaller cache and buffer, more heap
* use MAIN_EVENT_CLOCK_TICK to avoid audio glitches
* fix: esp_psram_get_size not found in c3
* Bump to 1.9.0
* fix: resolve crash when closing codec dev on esp-hi
* fix: fix incorrect status display in non-zh-CN languages
* fix: reduce noise when not in Speaking state
* Adapt for LilyGO-T-Circle-S3 device
* Adapt for LilyGO-T-Circle-S3 device
* Remove comments and modify the size of the lilygo-t-circle-s3 image
* Modify the code style and format to Google C++
* Modify the code style and format to Google C++
* Fixed bugs in the LILYGO T-Circle-S3 board and added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Merge branch 'main' of https://github.com/Llgok/xiaozhi-esp32
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Fix the color display issue for T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Update T-CameraPlus-S3_V1.2 Version Xiaozhi Example
* Resolve the issue where the camera on the T-CameraPlus-S3_V1.2 board cannot be used normally.
* Enhance microphone reception volume
* fix the issue where voice wake-up is not working
* fix the issue where voice wake-up is not working
* set camera HMirror to true
* change key1 to camera button
* change key1 to camera button
* change key1 to camera button
* add ir filter controller
* add ir filter controller
* add ir filter controller
* Added third-party hardware support
Waveshare-ESP32-P4-WIFI6-Touch-LCD-3.4C,
Waveshare-ESP32-P4-WIFI6-Touch-LCD-4C,
Waveshare-ESP32-P4-WIFI6-Touch-LCD-4B
* Remove the README
* Merge ESP32-P4-WIFI6-Touch-LCD-3.4C and ESP32-P4-WIFI6-Touch-LCD-4C into one board
* Add readme files
* Adapt for LilyGO-T-Circle-S3 device
* Adapt for LilyGO-T-Circle-S3 device
* Remove comments and modify the size of the lilygo-t-circle-s3 image
* Modify the code style and format to Google C++
* Modify the code style and format to Google C++
* Fixed bugs in the LILYGO T-Circle-S3 board and added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Merge branch 'main' of https://github.com/Llgok/xiaozhi-esp32
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Fix the color display issue for T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Update T-CameraPlus-S3_V1.2 Version Xiaozhi Example
* Adjust dependency components for ESP32-P4
* Add backlight control and touch control to Waveshare ESP32-P4-NANO
* Increase the amount of saved information for Waveshare ESP32-P4-NANO
* feat(ota): add logging for missing sections in Ota::CheckVersion
Add detailed logging to inform when specific sections (activation, mqtt, websocket, server_time, firmware) are not found during the OTA version check. This improves debugging and visibility into the OTA process.
* 恢复提示词
---------
Co-authored-by: loadingxv <loadingvx@gmail.com>
* Adapt for LilyGO-T-Circle-S3 device
* Adapt for LilyGO-T-Circle-S3 device
* Remove comments and modify the size of the lilygo-t-circle-s3 image
* Modify the code style and format to Google C++
* Modify the code style and format to Google C++
* Fixed bugs in the LILYGO T-Circle-S3 board and added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Merge branch 'main' of https://github.com/Llgok/xiaozhi-esp32
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Added support for two new boards: LILYGO T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* Fix the color display issue for T-Display-S3-Pro-MVSRLora and LILYGO T-Display-S3-Pro-MVSRLora_NO_BATTERY.
* The atk-dnesp32s3-box is compatible with ES8311 devices and non-ES8311 devices
* Update atk_dnesp32s3_box.cc
format code
---------
Co-authored-by: Xiaoxia <terrence@tenclass.com>
* fix typo, add missing prefix `CONFIG` to `ESP_TASK_WDT_TIMEOUT_S`
* add KNOB gpio spec to sensecap config
* create new knob component
* implement ajust output volume with knob
* modify function name to UpperCamelCase
* Tidy up comments and logs
* fix: fixed the issue that the device enters re-network configuration mode by mistake when long pressing to start the device.
* fix: Do not shut down when connected to type-c.
* fix: fixed the issue that firmware verification failed during OTA for 32M flash.
* doc: update readme.
* fix: Solve the problem that the partition table configuration is not effective.
* Add files via upload
Added board description
* Add files via upload
New battery control
* Add files via upload
New battery control
* Add files via upload
Update format
* Add files via upload
Added screen version compatibility
* Add files via upload
Added screen version compatibility
* Add files via upload
修改程序风格
* Add files via upload
修改程序风格
* Add files via upload
更新背光
* Add files via upload
Update the backlight control program
* Add files via upload
Update the backlight control program
* Add files via upload
Update the backlight control program
* Add files via upload
Update function function implementation
* Add files via upload
Update function function implementation
* Add files via upload
Remove excess space
* Add files via upload
添加注释
* Add files via upload
添加程序注释
- Add support for device activation with audio feedback
- Refactor OTA update flow to include activation code handling
- Update asset management for localized sound resources
- Improve error handling and device state management
- Reorganize binary asset includes and CMake configuration
* Adapt for LilyGO-T-Circle-S3 device
* Adapt for LilyGO-T-Circle-S3 device
* Remove comments and modify the size of the lilygo-t-circle-s3 image
* Modify the code style and format to Google C++
* Modify the code style and format to Google C++
---------
Co-authored-by: Xiaoxia <terrence@tenclass.com>
* Added ESP32-S3-Touch-AMOLED-1.8 support
* Add ESP32-S3-Touch-AMOLED-1.8 support README
* Change the default image format
---------
Co-authored-by: Xiaoxia <terrence@tenclass.com>
description:Report a reproducible problem setting up, building, or flashing the firmware
title:"[Build] "
labels:['bug']
body:
- type:markdown
attributes:
value:|
This form is for the open-source firmware. For device unbinding, email [xiaozhi.ai@tenclass.com](mailto:xiaozhi.ai@tenclass.com?subject=Device%20unbinding%20request) privately with the device ID and MAC address. Use the [XiaoZhi AI website](https://xiaozhi.me/) for other account and cloud-service requests.
- type:checkboxes
id:checklist
attributes:
label:Before submitting
options:
- label:I have tested a current source version and searched for an existing issue reporting the same problem.
required:true
- type:input
id:board
attributes:
label:Board or device
description:Enter the physical product/model and the selected board target or build variant.
description:Report a reproducible firmware problem while a device is running
title:"[Bug] "
labels:['bug']
body:
- type:markdown
attributes:
value:|
This form is for the open-source device firmware. For device unbinding, email [xiaozhi.ai@tenclass.com](mailto:xiaozhi.ai@tenclass.com?subject=Device%20unbinding%20request) privately with the device ID and MAC address. Use the [XiaoZhi AI website](https://xiaozhi.me/) for other account and cloud-service requests.
- type:checkboxes
id:checklist
attributes:
label:Before submitting
options:
- label:I have tested the latest applicable firmware and searched for an existing issue reporting the same problem.
required:true
- type:input
id:device
attributes:
label:Device or development board
description:Enter the manufacturer, exact product/model, and hardware revision or SKU when known.
placeholder:ex. LilyGO T-Circle-S3 V1.1 or EchoEar N16R8
validations:
required:true
- type:input
id:firmware_version
attributes:
label:Firmware version
description:Paste the `App version:` value from the startup log. Do not enter a board name or "latest".
placeholder:ex. 2.2.6
validations:
required:true
- type:input
id:firmware_source
attributes:
label:Firmware source or filename
description:State where the firmware came from and include its filename or board target when available.
description:Describe the starting state, exact actions, expected result, actual result, frequency, and any relevant modifications.
placeholder:|
Steps:
Expected result:
Actual result:
Frequency:
Modifications, if any:
validations:
required:true
- type:textarea
id:evidence
attributes:
label:Logs or other evidence
description:Paste relevant startup and failure logs, including a backtrace or reset reason when applicable. If logs are unavailable, briefly say why. Remove credentials, device IDs, MAC addresses, UUIDs, and other private information.
placeholder:Paste relevant logs here, or explain why logs cannot be collected.
render:shell
validations:
required:true
- type:upload
id:attachments
attributes:
label:Optional photos, videos, or log files
description:Attach files that help identify the hardware or demonstrate the failure. Review files for private information first.
description:Propose an improvement to a supported device or the common firmware
title:"[Feature] "
labels:['enhancement']
body:
- type:markdown
attributes:
value:|
Use the **Board support request** form for a new board, hardware revision, or product variant. Account and cloud-product requests belong on the [XiaoZhi AI website](https://xiaozhi.me/).
- type:checkboxes
id:checklist
attributes:
label:Before submitting
options:
- label:I have searched the existing feature requests and did not find the same proposal.
required:true
- type:input
id:scope
attributes:
label:Affected device or area
description:Enter the device, board target, or firmware area. Enter "All devices" for a general request.
placeholder:ex. df-k10, audio, display, protocol, or All devices
validations:
required:true
- type:textarea
id:problem
attributes:
label:Problem or use case
description:Explain the current limitation and why the requested change would be useful.
validations:
required:true
- type:textarea
id:solution
attributes:
label:Requested behavior
description:Describe the result or behavior you would like to see.
validations:
required:true
- type:textarea
id:context
attributes:
label:Additional context
description:Optionally add alternatives, screenshots, diagrams, links, or testing information.
description:Request support for a new board, product, hardware revision, or SKU
title:"[Board Support] "
labels:['enhancement']
body:
- type:markdown
attributes:
value:|
Similar-looking products may use different pins and components. Please identify the exact hardware rather than only naming the closest supported board.
- type:checkboxes
id:checklist
attributes:
label:Before submitting
options:
- label:I have searched the repository and existing issues for this exact model and hardware revision.
required:true
- type:textarea
id:hardware
attributes:
label:Exact hardware
description:Provide the manufacturer, model, revision/SKU, main chip and memory, and an official product or hardware documentation link.
placeholder:|
Manufacturer and model:
Revision or SKU:
Chip, flash, and PSRAM:
Official link:
validations:
required:true
- type:textarea
id:request
attributes:
label:Requested support
description:Describe what should work and any known differences from the closest supported board.
validations:
required:true
- type:textarea
id:attempted
attributes:
label:What have you tried?
description:Include the closest board target or firmware, its version, and the result. Enter "None" if no existing target is appropriate.
validations:
required:true
- type:dropdown
id:testing
attributes:
label:Testing available
options:
- I can build, flash, collect logs, and test changes
- I can flash binaries and test changes
- I can provide hardware information only
- I cannot test changes
validations:
required:true
- type:upload
id:attachments
attributes:
label:Optional hardware files, photos, or logs
description:Attach public schematics, board photos, pinouts, or serial logs. Do not upload confidential material or secrets.
description:Ask a focused technical question about this repository or supported hardware
title:"[Question] "
labels:['question']
body:
- type:markdown
attributes:
value:|
This form is for technical questions about the open-source firmware. For device unbinding, email [xiaozhi.ai@tenclass.com](mailto:xiaozhi.ai@tenclass.com?subject=Device%20unbinding%20request) privately with the device ID and MAC address. Other account and cloud-service requests belong on the [XiaoZhi AI website](https://xiaozhi.me/).
- type:checkboxes
id:checklist
attributes:
label:Before submitting
options:
- label:I have searched the repository and existing issues and did not find an answer.
required:true
- type:input
id:context
attributes:
label:Relevant device, version, or code area
description:Enter the device/model, board target, source or firmware version, file, or subsystem that matters to the question.
placeholder:ex. zhengchen-cam on v2.2.6, or main/audio on current main
validations:
required:true
- type:textarea
id:question
attributes:
label:Goal and question
description:Explain what you are trying to accomplish and state the specific technical question.
validations:
required:true
- type:textarea
id:attempted
attributes:
label:What have you tried?
description:Optionally include relevant files, commands, experiments, logs, screenshots, or a minimal example.
This GitHub repository tracks the open-source XiaoZhi device firmware. Account recovery, device ownership, and cloud-service requests are handled outside the public issue tracker.
## Unbind a Previously Owned Device
If you purchased or received a used device that is still bound to another person's account:
1. Collect the device ID and MAC address using the instructions below.
2. Email [xiaozhi.ai@tenclass.com](mailto:xiaozhi.ai@tenclass.com?subject=Device%20unbinding%20request) with both identifiers.
3. Use an email subject such as `Device unbinding / 解绑设备 - Device ID XXX - MAC address XXX`.
For multiple devices, attach a list containing the device ID and MAC address for each device.
> **Privacy:** Send device IDs and MAC addresses only by email. Do not post them in a public GitHub issue, discussion, screenshot, or log.
### Why Both Identifiers Are Required
- The **MAC address** uniquely identifies the physical device hardware.
- The **device ID** is obtained by asking the AI running on the device. Providing the value reported by the device helps demonstrate that the requester has the device in hand and can operate it.
Together, these identifiers help the support team locate the correct binding record and avoid unbinding the wrong device.
### Find the Device ID
If the device can connect and have a conversation, ask it:
> What is my device ID?
Record the complete value reported by the device. Both the device ID and MAC address are required for the standard unbinding process. If the device cannot have a conversation or does not return a device ID, explain this in the email; the support team may require other proof of possession. Do not guess or substitute a temporary activation code.
### Find the MAC Address
1. Connect the powered-on device to a computer with a data-capable USB cable.
2. Open the device's serial port with a serial terminal. Developers with an ESP-IDF environment can use `idf.py monitor`.
3. Keep the serial terminal open and restart the device so that the complete startup log is captured.
4. Search the log for a line similar to:
```text
wifi:mode : sta (aa:bb:cc:dd:ee:ff)
```
5. The value inside the parentheses is the Wi-Fi MAC address to include in the email.
If that line does not appear, search the startup log for `MAC` or `mac_address`. If you still cannot identify it, attach the exact device model and explain the situation in the private email. Do not upload the unredacted startup log to a public GitHub issue because it may contain the MAC address, UUID, Wi-Fi name, and other device information.
## Other Account and Cloud-Service Requests
For verification codes, password recovery, activation, agent configuration, voiceprint, voice cloning, and other cloud services, use the [XiaoZhi AI website](https://xiaozhi.me/).
XiaoZhi is an ESP-IDF C/C++ voice-assistant firmware supporting many chips, boards, displays, audio devices, and network transports. A build selects exactly one board implementation.
Use ESP-IDF v6.0.2 when possible. IDF 5.5.x is retained only for documented legacy boards.
## Architecture
-`main/application.*`: main event loop, protocol lifecycle, and high-level behavior.
-`main/device_state_machine.*`: legal runtime state transitions.
-`main/boards/common/`: board interfaces and reusable hardware/network helpers.
-`main/boards/**/`: board-specific pins, initialization, and build variants.
-`main/audio/`: codecs, audio tasks, engines, wake words, and queues.
-`main/protocols/`: transport-neutral API plus WebSocket and MQTT/UDP.
-`main/display/` and `main/led/`: reusable UI implementations.
-`main/mcp_server.*`: common device-side MCP tools and dispatch.
-`main/Kconfig.projbuild`: board and feature configuration.
-`main/CMakeLists.txt`: source, board, locale, font, and asset selection.
Read the closest existing implementation before adding a new one. Prefer the narrowest owning layer; do not put board-specific behavior into core modules.
## Required Rules
- Preserve unrelated worktree changes and keep patches focused.
- A build must export exactly one board factory through `DECLARE_BOARD(...)`.
- Never alter an existing board's pins to support different hardware. Add a uniquely named board or release variant; board identity affects OTA compatibility.
- Core code depends on `Board` interfaces, never a concrete board class or board `config.h`.
- Treat camera, backlight, display, LED, battery, and similar capabilities as optional.
- Change runtime state through `Application::SetDeviceState()` and the state machine.
- Callbacks may run outside the main task. Schedule application mutations with `Application::Schedule()` or event bits.
- Do not block the main event loop or audio tasks. Avoid unbounded queues and repeated large allocations in audio paths.
- Keep shared message semantics in `Protocol`; verify both transports when changing its contract.
- Validate network input and preserve `cJSON` ownership. NVS keys are persistent API and require migration when changed.
- Guard target-specific features with Kconfig/component rules. Do not assume every target has PSRAM or S3/P4 resources.
- Do not manually edit generated/vendor output: `build/`, `releases/`, `managed_components/`, `components/`, `sdkconfig*`, `main/assets/lang_config.h`, or generated mmap headers.
- Format only touched C/C++ files with the repository `.clang-format`; avoid unrelated mass formatting.
When adding a board or variant, update every relevant link in that chain. Include a unique board identity, correct chip target, flash/partition settings, exactly one `DECLARE_BOARD`, and board documentation. Follow `docs/custom-board.md`.
[手工打造你的 AI 女友,新手入门教程【bilibili】](https://www.bilibili.com/video/BV1XnmFYLEJN/)
👉 [Human: Give AI a camera vs AI: Instantly finds out the owner hasn't washed hair for three days【bilibili】](https://www.bilibili.com/video/BV1bpjgzKEhd/)
## 项目目的
👉 [Handcraft your AI girlfriend, beginner's guide【bilibili】](https://www.bilibili.com/video/BV1XnmFYLEJN/)
本项目基于乐鑫的 ESP-IDF 进行开发。
As a voice interaction entry, the XiaoZhi AI chatbot leverages the AI capabilities of large models like Qwen / DeepSeek, and achieves multi-terminal control via the MCP protocol.
本项目是一个开源项目,主要用于教学目的。我们希望通过这个项目,能够帮助更多人入门 AI 硬件开发,了解如何将当下飞速发展的大语言模型应用到实际的硬件设备中。无论你是对 AI 感兴趣的学生,还是想要探索新技术的开发者,都可以通过这个项目获得宝贵的学习经验。
<img src="docs/mcp-based-graph.jpg" alt="Control everything via MCP" width="320">
欢迎所有人参与到项目的开发和改进中来。如果你有任何想法或建议,请随时提出 issue 或加入群聊。
## Recent Updates
学习交流 QQ 群:946599635
- The mainline now targets ESP-IDF v6.0 or later, with v6.0.2 as the preferred stable SDK. The previous 157-variant baseline was validated on ESP-IDF v6.0.1; the current matrix contains 171 variants, of which 170 support IDF 6.0.x and the ESP32-S31 variant requires IDF 6.1 or later.
- MQTT and BluFi cryptographic code has migrated to PSA Crypto. IDF 6 component splits and third-party dependency compatibility have also been addressed.
- Audio pipeline concurrency, MQTT/UDP packet validation, and release-matrix selection have been hardened.
- ESP-IDF v5.5 is retained only for documented legacy boards. ESP32-P4 Rev1 and Rev3 are both supported on IDF 6 with ESP-SR 2.4.7; see the [ESP-IDF 6.0 Migration Guide](docs/esp-idf-6-migration.md) for full compatibility and board-validation details.
## 已实现功能
### Features Implemented
- Wi-Fi 配网
-支持 BOOT 键唤醒和打断
-离线语音唤醒(乐鑫方案)
-流式语音对话(WebSocket 或 UDP 协议)
-支持国语、粤语、英语、日语、韩语 5 种语言识别(SenseVoice 方案)
-声纹识别(识别是谁在喊 AI 的名字,[3D Speaker 项目](https://github.com/modelscope/3D-Speaker))
-使用大模型 TTS(火山引擎与 CosyVoice 方案)
-支持可配置的提示词和音色(自定义角色)
-Qwen2.5 72B 或 豆包 API
-支持每轮对话后自我总结,生成记忆体
-扩展液晶显示屏,显示信号强弱
-支持 ML307 Cat.1 4G 模块
- Wi-Fi, wired Ethernet, USB RNDIS, and ML307/EC801E or NT26 Cat.1 4G networking; supported boards can switch between Wi-Fi and 4G
-Offline voice wake-up with [ESP-SR](https://github.com/espressif/esp-sr), including customizable wake words
-Two communication transports: [WebSocket](docs/websocket.md) and [MQTT + UDP](docs/mqtt-udp.md)
-Opus audio streaming with conventional streaming ASR + LLM + TTS pipelines and Realtime end-to-end voice models; AEC-capable hardware supports realtime full-duplex interaction
-Speaker recognition, identifies the current speaker [3D Speaker](https://github.com/modelscope/3D-Speaker)
-OLED / LCD displays with emoji and rich expression support, plus camera vision input on supported boards
-Battery display and power management
-38 interface languages, with localized voice prompts where available and English fallback
-ESP32, ESP32-C3, ESP32-C5, ESP32-C6, ESP32-S3, and ESP32-P4 chip platforms
-Wi-Fi provisioning through hotspot or BluFi
-Device-side MCP for device control (Speaker, LED, Servo, GPIO, etc.)
-Cloud-side MCP to extend large model capabilities (smart home control, PC desktop operation, knowledge search, email, etc.)
- Customizable wake words, fonts, emojis, and chat backgrounds with online web-based editing ([Custom Assets Generator](https://github.com/78/xiaozhi-assets-generator))
## 硬件部分
## Hardware
为方便协作,目前所有硬件资料都放在飞书文档中:
### Breadboard DIY Practice
[《小智 AI 聊天机器人百科全书》](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb?from=from_copylink)
See the Feishu document tutorial:
面包板接线图如下:
👉 ["XiaoZhi AI Chatbot Encyclopedia"](https://ccnphfhqs21z.feishu.cn/wiki/F5krwD16viZoF0kKkvDcrZNYnhb?from=from_copylink)
For beginners, it is recommended to use the firmware that can be flashed without setting up a development environment.
The firmware connects to the official [xiaozhi.me](https://xiaozhi.me) server by default. Personal users can register an account to use the Qwen real-time model for free.
- Install the ESP-IDF plugin. [ESP-IDF v6.0.2](https://github.com/espressif/esp-idf/releases/tag/v6.0.2) is preferred; use a stable v6.0 or later release. ESP-IDF v5.5.2 is retained only for legacy board compatibility
- Linux is better than Windows for faster compilation and fewer driver issues
- This project uses Google C++ code style, please ensure compliance when submitting code
### Developer Documentation
- [ESP-IDF 6.0 Migration Guide](docs/esp-idf-6-migration.md) - SDK compatibility, component changes, legacy hardware support, and board validation status
- [Custom Board Guide](docs/custom-board.md) - Learn how to create custom boards for XiaoZhi AI
- [MCP Protocol IoT Control Usage](docs/mcp-usage.md) - Learn how to control IoT devices via MCP protocol
- [MQTT + UDP Hybrid Communication Protocol Document](docs/mqtt-udp.md)
- [A detailed WebSocket communication protocol document](docs/websocket.md)
## Large Model Configuration
If you already have a XiaoZhi AI chatbot device and have connected to the official server, you can log in to the [xiaozhi.me](https://xiaozhi.me) console for configuration.
👉 [Backend Operation Video Tutorial (Old Interface)](https://www.bilibili.com/video/BV1jUCUY2EKM/)
## Related Open Source Projects
For server deployment on personal computers, refer to the following open-source projects:
- [xinnan-tech/xiaozhi-esp32-server](https://github.com/xinnan-tech/xiaozhi-esp32-server) Python server
- [joey-zhou/xiaozhi-esp32-server-java](https://github.com/joey-zhou/xiaozhi-esp32-server-java) Java server
- [AnimeAIChat/xiaozhi-server-go](https://github.com/AnimeAIChat/xiaozhi-server-go) Golang server
- [hackers365/xiaozhi-esp32-server-golang](https://github.com/hackers365/xiaozhi-esp32-server-golang) Golang server
Other client projects using the XiaoZhi communication protocol:
This document explains how to enable and use BluFi (BLE-based WiFi provisioning) in the XiaoZhi firmware, together with the in-tree `esp-wifi-connect` component that handles WiFi connection and credential storage. See the official [Espressif BluFi documentation](https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-guides/ble/blufi.html) for the protocol details.
## Prerequisites
- A chip and firmware configuration that support BLE.
- In `idf.py menuconfig`, enable `WiFi Configuration Method -> ESP-BluFi` (`CONFIG_USE_ESP_BLUFI_WIFI_PROVISIONING=y`). If you want to use BluFi, disable the Hotspot option in the same menu; otherwise hotspot provisioning wins by default.
- Keep the default NVS and event-loop initialization provided by the project's `app_main`.
- Exactly one of `CONFIG_BT_BLUEDROID_ENABLED` / `CONFIG_BT_NIMBLE_ENABLED` must be selected; they are mutually exclusive.
## Workflow
1. A phone (using the official EspBlufi app or another BluFi client) connects to the device over BLE and sends the target WiFi SSID / password. The phone can also request the list of WiFi networks scanned by the device through the BluFi protocol.
2. In `ESP_BLUFI_EVENT_REQ_CONNECT_TO_AP`, the device stores the credentials into `SsidManager` (persisted in NVS by the `esp-wifi-connect` component).
3. The device then launches `WifiStation` to scan and connect; progress is reported back over BluFi.
4. If provisioning succeeds, the device connects to the new WiFi automatically. If it fails, an error status is sent back.
## Steps
1.**Configure**: turn on `ESP-BluFi` in menuconfig, then build and flash the firmware.
2.**Trigger provisioning**: at first boot with no stored WiFi credentials the device enters provisioning automatically.
3.**Phone side**: open the EspBlufi app (or another BluFi client), scan and connect to the device, optionally enable encryption, then enter the WiFi SSID / password and send them.
4.**Observe the result**:
- Success: BluFi reports success and the device connects to WiFi.
- Failure: BluFi reports failure; retry or check the router.
## Notes
- BluFi cannot be used at the same time as hotspot provisioning. If hotspot provisioning is already enabled, the device will use it. Keep only one provisioning method in menuconfig.
- When running repeated tests, clear or overwrite the stored SSID (`wifi` NVS namespace) to avoid stale credentials interfering with the next run.
- If you write your own BluFi client, follow the official protocol frame format linked above.
- The EspBlufi app download links are listed in the official documentation.
- Because the BluFi API changed in IDF 5.5.2, firmware built with 5.5.2 advertises the Bluetooth name as `"Xiaozhi-Blufi"`, while 5.5.1 uses `"BLUFI_DEVICE"`.
This project uses `clang-format` to keep the code style consistent. The `.clang-format` file in the project root is based on the Google C++ style guide with a few project-specific tweaks.
### Installing clang-format
Make sure `clang-format` is available before you use it:
This guide describes how to add a new board to the XiaoZhi AI voice assistant project. XiaoZhi AI supports 70+ ESP32-series boards; each one lives in its own directory under `main/boards/`.
## Important
> **Warning**: for a custom board whose IO configuration differs from an existing board, never overwrite the original board's configuration. Always create a new board type - or use the `builds` array in `config.json` to produce a distinct firmware name with different `sdkconfig` macros. Use `python scripts/build.py [board-directory]` to build the firmware.
>
> Overwriting an existing board's configuration is dangerous because OTA updates may replace your custom firmware with the stock firmware for the original board. Every board must have a unique identity and its own firmware update channel.
## Directory Layout
A board directory typically contains:
-`xxx_board.cc` - board-level initialization and glue code.
-`config.h` - pin assignments and board-level settings.
-`config.json` - reported board type and release configuration consumed by CMake and `scripts/build.py`.
-`README.md` - board-specific notes.
Boards can live directly under `main/boards/` or be grouped by manufacturer under `main/boards/<manufacturer>/<board>/` (see [Manufacturer Sub-directories](#manufacturer-sub-directories) below).
## Steps
### 1. Create the Board Directory
Create a new directory under `main/boards/` using the `[vendor]-[model]` naming style (e.g. `m5stack-tab5`):
- Reads `target` from `config.json`. It calls `idf.py set-target` only when the
target changes, then regenerates `sdkconfig` from defaults and the selected
build's `sdkconfig_append`.
- Passes the selected build's `name` as the reported firmware variant name.
- Builds `build/merged-binary.bin` without creating a ZIP by default. Pass
`--zip` to recreate `releases/v<version>_<name>.zip`.
### 6. Write the README
In `README.md`, describe the board, hardware requirements, build instructions, and any special notes.
## Manufacturer Sub-directories
Boards can be grouped by manufacturer under `main/boards/<manufacturer>/<board>/`. This is the recommended layout when a single vendor ships several variants - for example `main/boards/waveshare/esp32-p4-nano/` or `main/boards/lceda-course-examples/eda-tv-pro/`.
For a board in a manufacturer sub-directory, add the same value to `config.json`, for example `"manufacturer": "waveshare"`. The firmware reports it as `board.manufacturer` together with `board.type` and `board.name`. Flat community boards without this field report an empty manufacturer string.
Set `BOARD_DIR` to the complete path relative to `main/boards/`:
```cmake
elseif(CONFIG_BOARD_TYPE_WAVESHARE_ESP32_P4_NANO)
set(BOARD_DIR "waveshare/esp32-p4-nano")
set(BUILTIN_TEXT_FONT font_puhui_basic_30_4)
set(BUILTIN_ICON_FONT font_awesome_30_4)
set(DEFAULT_EMOJI_COLLECTION twemoji_64)
```
The build system loads sources from `main/boards/${BOARD_DIR}/` and reads the reported board type from that directory's `config.json`. If `config.json` or its top-level `type` is absent, the full `BOARD_DIR` with `/` replaced by `-` is used as the fallback type.
Rules of thumb:
- Use the manufacturer layout when you have two or more boards from the same vendor that share drivers, assets, or documentation.
- Use the flat layout for one-off boards and community examples.
- Directory names use lowercase with dashes (e.g. `waveshare`, `lceda-course-examples`).
## Common Board Components
Several reusable components live in `main/boards/common/`. You can include them directly from your board class:
### Display drivers
Supported LCD families include:
- ST7789 (SPI)
- ILI9341 (SPI)
- SH8601 (QSPI)
- and many more.
### Audio codecs
- `Es8311AudioCodec` (most common)
- `Es8374AudioCodec`
- `Es8388AudioCodec`
- `Es8389AudioCodec`
- `BoxAudioCodec` (ES7210 mic array + codec combo used on ESP-Box boards)
- `NoAudioCodec` (direct I2S without external codec)
- `DummyAudioCodec` (placeholder for boards without audio)
### Power management
- `Axp2101` power management IC helpers.
- `Sy6970` battery charger helpers.
- `AdcBatteryMonitor` - simple ADC-based battery voltage monitor.
- `PowerSaveTimer` / `SleepTimer` - helpers for light-sleep scheduling.
### Networking
- `WifiBoard` - WiFi-only base class.
- `Ml307Board` / `Nt26Board` - 4G modem base classes.
- `DualNetworkBoard` - switchable WiFi / 4G base class.
- `EspVideo` helpers for ESP-Video on ESP32-S3 / ESP32-P4.
### Input helpers
- `Button` - standard push buttons (click, long-press, multi-click).
- `Knob` - rotary encoder wrapper.
- `PressToTalkMcpTool` - push-to-talk tool that registers itself through MCP.
- `SystemReset` - helper that performs a safe factory reset when a button is held at boot.
### MCP integration
Any board can register custom tools - speaker control, screen brightness, battery readout, light control, etc. See [MCP IoT control usage](./mcp-usage.md).
## Board Class Hierarchy
- `Board` - base class
- `WifiBoard` - WiFi-connected board
- `Ml307Board` / `Nt26Board` - 4G modem boards
- `DualNetworkBoard` - WiFi + 4G switchable board
- `RndisBoard` - RNDIS-over-USB board
## Tips
1. **Start from a similar board** - copying and tweaking an existing board is usually faster than starting from scratch.
2. **Bring up incrementally** - get the display up first, then audio, then the full stack.
3. **Double check pin assignments** - every pin defined in `config.h` must match your schematic.
# ESP-IDF 6.0 Migration and Board Compatibility Status
> Last updated: 2026-07-24
> Validated SDK: ESP-IDF v6.0.1
> Scope: 138 board directories and 171 supported build variants defined by `main/boards/**/config.json`.
## Current Status
The current release matrix contains 171 variants: 170 select on IDF 6.0.x, while the ESP32-S31 variant requires IDF 6.1 or later. The previous 157-variant baseline passed the latest complete GitHub Actions matrix on ESP-IDF 6.0.1. ESP-SR 2.4.7 adds IDF 6 support for ESP32-P4 Rev < 3, so the 14 unsuffixed Rev1 variants now join their existing `-p4x` Rev3 counterparts in the IDF 6 matrix. Component versions that use ranges are resolved from the Component Registry at build time; until per-target lock snapshots are committed, this is a source-reproducible build rather than a bit-for-bit dependency-reproducible build.
| Status | Variants | Meaning |
|---|---:|---|
| ✅ Previously matrix-validated | 157 | All 157 baseline builds completed in GitHub Actions with ESP-IDF 6.0.1; this does not imply hardware or complete peripheral validation |
| 🟡 Net variants added since full-matrix validation | 14 | Includes the 14 newly enabled P4 Rev1 variants and the IDF 6.1-only ESP32-S31 variant, after retired low-use variants were removed; a complete current-matrix CI run remains pending |
| 🟡 Feature-degraded subset | 1 | `esp-vocat` builds on IDF 6, but its PCB capacitive slider/button support is disabled pending compatible touch-sensor components |
| 🔴 Build-blocked | 0 | No supported release variant remains blocked at compile or link time |
There are no known remaining IDF 6 build blockers. [`78/esp_lcd_nv3023 1.0.1`](https://components.espressif.com/components/78/esp_lcd_nv3023/versions/1.0.1) and [`wvirgil123/sscma_client 1.0.3`](https://components.espressif.com/components/wvirgil123/sscma_client/versions/1.0.3/readme) are consumed directly from the Component Registry, so local copies under the ignored `components/` directory are not required. ESP32-P4 Rev1 and Rev3 now share the IDF 6 build path while retaining separate artifact names and silicon-selection settings.
Representative local full-build results are shown below. Firmware size and free space are reported by ESP-IDF 6.0.1 `check_sizes.py`. The authoritative per-variant compatibility result is the full GitHub Actions matrix in the next section.
| Chip | Build variant | Application size | Free space in smallest app partition |
The old negative result for `esp32-p4-function-ev-board` came from ESP-SR 2.4.6. ESP-SR 2.4.7 supplies the missing ESP32-P4 Rev < 3 libraries for IDF 6, so the project no longer version-gates the P4 variants.
For backward-compatibility regression coverage, `xmini-c3` also completed a full build with ESP-IDF 5.5.4 (application size `0x234920`, 44% free in the smallest app partition). The legacy `esp32-p4-function-ev-board` release variant subsequently completed the same 5.5.4 release flow (application size `0x38a130`, 10% free), including merged-binary packaging. This confirms that the compatibility changes for I2S port numbering, LCD I2C configuration, the UHCI DMA dependency, and the pre-v3 P4 selection did not break the existing 5.5 build chain.
## Full-Matrix CI Validation
GitHub Actions run [29534954031](https://github.com/78/xiaozhi-esp32/actions/runs/29534954031) built the then-current matrix with the `espressif/idf:v6.0.1` container. The matrix-generation job and all 157 board builds passed. This historical run predates the current 170-variant IDF 6.0.x matrix.
Results by chip target:
| Chip target | Variants | Passed | Blocked |
|---|---:|---:|---:|
| ESP32 | 7 | 7 | 0 |
| ESP32-C3 | 9 | 9 | 0 |
| ESP32-C5 | 4 | 4 | 0 |
| ESP32-C6 | 9 | 9 | 0 |
| ESP32-S3 | 114 | 114 | 0 |
| ESP32-P4 v3.x | 14 | 14 | 0 |
| **Total** | **157** | **157** | **0** |
`esp-vocat` remains feature-degraded because IDF 6 builds omit the PCB capacitive slider/button path; its CST816 display touch remains enabled.
## ESP32-P4 Silicon Scope and Naming
The IDF 6 release matrix supports both ESP32-P4 Rev < 3 and Rev >= 3 silicon. The two silicon families keep distinct artifact names and sdkconfig settings:
| SDK | ESP32-P4 Rev < 3 | ESP32-P4 Rev >= 3 |
|---|---|---|
| ESP-IDF < 6 | Original `esp32-p4-` name; adds `CONFIG_ESP32P4_SELECTS_REV_LESS_V3=y` and `CONFIG_ESP32P4_REV_MIN_100=y` | Uses `esp32-p4x-` when the variant starts with the chip name; otherwise retains the established `-p4x` suffix |
| ESP-IDF >= 6 | Original `esp32-p4-` name; adds `CONFIG_ESP32P4_SELECTS_REV_LESS_V3=y` and `CONFIG_ESP32P4_REV_MIN_100=y` | Uses `esp32-p4x-` when the variant starts with the chip name; otherwise retains the established `-p4x` suffix |
ESP-IDF 5.5 and 6.0.x each select 170 variants: 14 P4X artifacts, 14 Rev < 3 P4 artifacts with their original names, and 142 non-P4 variants. ESP-IDF 6.1 or later additionally selects the ESP32-S31 build, for 171 variants in total.
Espressif's current chip-identification table lists v0.0, v1.0, v1.3, v3.0, v3.1, and v3.2, with no v4.0 revision. The public errata history added v3.0/v3.1 information on 2026-02-12 and v3.2 information on 2026-04-20. ESP-IDF 6.0 release notes explicitly describe support for "ESP32-P4 Version3 silicon." See the official [chip revision identification](https://docs.espressif.com/projects/esp-chip-errata/en/latest/esp32p4/01-chip-identification/index.html), [errata revision history](https://docs.espressif.com/projects/esp-chip-errata/en/latest/esp32p4/revision-history/index.html), and [ESP-IDF releases](https://github.com/espressif/esp-idf/releases).
## Component Compatibility
| Component/module | Version or approach for IDF 6 | Status | Notes |
|---|---|---|---|
| `78/uart-uhci` | `0.3.2` | ✅ Upstream support | The registry release compiles under IDF 6.0.1 in GitHub Actions |
| `78/uart-eth-modem` | `0.6.0` | ✅ Upstream support | Pinned because it has the required RF-test event API while supporting both ESP-IDF 5.5.2+ and 6.0.1; version 0.6.1 requires IDF 6.0.1+ |
| `espressif/mqtt` | `1.0.0` | ✅ Upstream support | MQTT moved from a built-in SDK component to a Component Manager dependency in IDF 6 |
| `78/esp-ml307` | `3.6.6` + project-level `espressif/mqtt` | ✅ Builds in CI | The upstream source compiles under IDF 6.0.1 when the project supplies the MQTT component moved out of IDF |
| `espressif/esp_hosted` / `esp_wifi_remote` | `2.12.11` / `1.6.2` | ✅ Upstream support | Used for ESP32-P4 Hosted Wi-Fi |
| `espressif/esp_video` | `^2.0.1` | ✅ Upstream support | Currently resolves to 2.3.0 on S3 and 2.0.1 on P4 due to BSP constraints |
| `espressif/esp_lcd_st77916` | `2.0.2` | ✅ Upstream support | Major version 2 uses the IDF 6 panel I/O definitions |
| `espressif/esp_lcd_spd2010` | `^2.0.0` (resolved `2.0.0~1`) | ✅ Upstream support | Major version 2 declares ESP-IDF 6 compatibility |
| `espressif/esp_lcd_co5300` | `2.1.0` | ✅ Upstream support | Used with explicit IDF 6-compatible QSPI I/O configuration in board code |
| `esp_emote_expression` | `1.0.2` | ✅ Upstream support | Replaces the previous dependency on the built-in `json` component |
| `wvirgil123/sscma_client` | [`1.0.3`](https://components.espressif.com/components/wvirgil123/sscma_client/versions/1.0.3/readme) | ✅ Upstream support | Uses `espressif/cjson` and the split driver components on IDF 6 while retaining the legacy component names on older IDF releases |
| `espressif/servo` / `espfriends/servo_dog_ctrl` | `1.0.0` / `0.2.0` | ✅ Upstream support | Uses official registry releases; compilation and linking were validated in the full IDF 6.0.1 build of `esp-hi`, with no local override required |
| `78/esp_lcd_nv3023` | [`1.0.1`](https://components.espressif.com/components/78/esp_lcd_nv3023/versions/1.0.1) | ✅ Upstream-based registry release | Mirrors MakerM0 upstream commit `15dae953`; adds IDF 6 color-order and GPIO compatibility while retaining older IDF branches; validated through the registry in a clean `magiclick-c3` full build |
| `llgok/cpp_bus_driver` | Excluded from the IDF 6 baseline | ⚪ Waiting for upstream | Version `2.1.0` still requires local patches for IDF 6. It is used only by the `LILYGO T-Display-P4` source, and that board has no release `config.json`, so the component has been removed until upstream support is available |
| MQTT protocol AES-CTR | PSA Crypto | ✅ Ported | Replaces the legacy AES context API removed by IDF 6 / Mbed TLS 4 |
| BluFi security negotiation (conditional path) | PSA FFDH + SHA-256 + AES-CTR | ✅ Ported | Uses the ESP-IDF 6 security scheme with ffdhe3072 and passed local full ESP32-S3 builds with BluFi enabled on IDF 5.5.4 and 6.0.1. Legacy 1024-bit BluFi clients are not compatible and must be upgraded |
| `espressif/bmi270_sensor` | [`0.1.2`](https://components.espressif.com/components/espressif/bmi270_sensor/versions/0.1.2/readme?language=en) | ✅ Upstream support | Provides IDF 6.0 prebuilt libraries for ESP32-C5 and ESP32-S3; validated by a full `esp-spot-c5` build |
| `touch_slider_sensor` / `touch_button_sensor` | Disabled for IDF 6 | 🟡 Feature gap | Their manifests require IDF < 6.0, so ESP Vocat omits its PCB capacitive slider/button path on IDF 6. CST816 display touch is unaffected |
| ESP32-P4 Rev < 3 / `espressif/esp-sr` | `~2.4.7` | ✅ Upstream support | The 14 Rev1 variants are available on IDF 5.5 and IDF 6; P4X-specific names continue to select Rev >= 3 |
## Per-Board Progress
In the table below, "Board" is the source directory and "Build variant" is the firmware name used by the build script. Different chip revisions of the same board may have different status, so they are split into separate rows when necessary.
| Chip | Board | Build variant | IDF 6.0 status | Current validation | Blocker/next step |
1. Run the IDF 6 BluFi CI job, then hardware-test at least one P4 v1.3 device on the legacy SDK path. When IDF 6-compatible releases of `touch_slider_sensor` and `touch_button_sensor` become available, re-enable and hardware-test the ESP Vocat PCB capacitive slider/button path.
2. For every green variant, complete a minimal hardware smoke test covering boot, networking, audio input/output, display/touch when present, camera when present, and 4G/Ethernet when present.
3. Perform a physical negative test for the `xmini-c3`/`xmini-c3-v3` firmware guard. CI proves that both images compile; it does not prove that a wrong image is safely rejected. Acceptance requires flashing each wrong image to a sacrificial or recoverable board and verifying that startup stops before any board-specific power or peripheral initialization can cause damage.
4. Keep third-party experiments under the ignored `components/` directory out of the migration branch. The reproducible baseline must use the published `78/esp_lcd_nv3023 1.0.1` and `wvirgil123/sscma_client 1.0.3` packages. `espfriends/servo_dog_ctrl 0.2.0` is an upstream registry dependency and needs no local override.
## Reproduction
```bash
source ~/.espressif/v6.0.1/esp-idf/export.sh
python scripts/build.py <board> --name <variant>
```
This document is a migration-status snapshot. Build compatibility and hardware compatibility are tracked separately. Add "hardware validated" to a board only after validation on physical hardware is complete.
This document defines version 1 of the `glyph_push` protocol extension. The extension lets a server
send bitmap glyphs that are missing from a device's installed text font. It applies equally to the
WebSocket and MQTT/UDP transports because capability advertisement and incoming JSON handling are
implemented in the shared protocol layer.
The extension supplements text rendering only. It does not change the text, TTS audio, or STT
semantics of the containing message.
## 1. Capability advertisement
The device advertises support in its client `hello` message:
```json
{
"type":"hello",
"version":1,
"features":{
"mcp":true,
"glyph_push":true
},
"text_font":{
"bundle":"noto-v1",
"charset":"common",
"size":20,
"bpp":4
}
}
```
`features.glyph_push` indicates support for this extension. A server must treat a missing or false
value as unsupported. The `v` field in each pushed payload carries the extension version.
The `text_font` object describes the exact font data installed on the device:
| Field | Type | Meaning |
|---|---|---|
| `bundle` | string | Explicit font bundle identifier. It changes when glyph metrics, rendering behavior, character sets, or wire compatibility change. |
| `charset` | string | Installed character set. Version 1 devices report `basic` or `common`. |
| `size` | number | Text font pixel profile used by the firmware. |
| `bpp` | number | Bits per pixel of the text font bitmap, currently `1` or `4`. |
`basic` is the font linked into the firmware. The standard XiaoZhi assets report `common` after
loading their common font from the assets partition. The server must use the values from each
device's hello message rather than inferring them from the board model.
An OTA assets package may replace the text font with a different size, bpp, character set, or font
family. The firmware still loads any structurally valid CBIN font. When the package also provides
complete `text_font_meta` fields, the device advertises those active runtime values and validates
glyph pushes against them. A legacy or custom package without compatible glyph metadata continues
to use its custom font, emoji, colors, and background, but advertises `glyph_push: false` and omits
`text_font`. This prevents incompatible fallback glyphs without restricting theme customization.
## 2. Server glyph payload
The server may attach a `glyph_push` object to either of these server-to-device messages:
- a TTS message with `"type": "tts"` and `"state": "sentence_start"`;
- an STT message with `"type": "stt"`.
Example:
```json
{
"type":"tts",
"state":"sentence_start",
"text":"𠮷野家",
"glyph_push":{
"v":1,
"bundle":"noto-v1",
"size":20,
"bpp":4,
"glyphs":[
{
"codepoint":134071,
"adv_w":320,
"box_w":20,
"box_h":20,
"ofs_x":0,
"ofs_y":0,
"bitmap":"<base64-encoded bitmap>"
}
]
}
}
```
The payload header must match the device capability exactly:
| Field | Requirement |
|---|---|
| `v` | Must be `1`. |
| `bundle` | Must equal `text_font.bundle`. |
| `size` | Must equal `text_font.size`. |
| `bpp` | Must equal `text_font.bpp`. |
| `glyphs` | The partial glyph batch being pushed, containing at most 64 entries. |
Each item uses the LVGL native bitmap-font metrics:
| Field | Meaning |
|---|---|
| `codepoint` | Unicode code point from `1` through `0x10FFFF`. |
| `adv_w` | Horizontal advance in LVGL fixed-point units with four fractional bits (one pixel is 16 units). |
| `box_w`, `box_h` | Bitmap dimensions in pixels. Each dimension must be from 0 through 64. |
| `ofs_x`, `ofs_y` | Signed 16-bit glyph offsets relative to the text baseline and cursor position. |
| `bitmap` | Base64 encoding of the uncompressed LVGL plain bitmap. |
The decoded bitmap length must be exactly:
```text
ceil(box_w * box_h * bpp / 8)
```
The bitmap must use the same plain, zero-stride layout as the matching Noto full-bundle CBIN font.
Servers should extract and forward the bitmap and metrics directly from that CBIN profile instead of
rasterizing an unrelated font at request time.
The sum of decoded bitmap lengths in one payload must not exceed 64 KiB. If any header, glyph, or
bitmap is invalid, the device rejects the entire glyph payload but still displays the message text
using its installed fonts. A PSRAM device may also use fallback glyphs cached by earlier messages.
## 3. Server selection algorithm
For every connection that advertises `glyph_push: true`, the server should:
1. Resolve the full font bundle identified by `text_font.bundle`.
2. Select the CBIN profile matching `text_font.size` and `text_font.bpp`.
3. Decode the message text into Unicode code points.
4. Remove control characters, duplicates, and code points already present in
`text_font.charset`.
5. Extract the remaining glyphs from the full bundle.
6. Apply the per-message limits and attach one `glyph_push` object to the text message.
7. Omit `glyph_push` when no missing glyph is available.
The installed text font is searched before the dynamic fallback font. Pushed glyphs therefore fill
missing code points; they do not override glyphs in `basic` or `common`.
The full font bundle may be shared by all device connections in a server process. Per-connection
work is limited to using the capability tuple `(bundle, charset, size, bpp)` to select which glyphs
are missing and which profile to read.
## 4. Device cache behavior
All glyphs in one message are inserted first, followed by a single fallback-font rebuild. The device
never rebuilds the font once per glyph.
On a device with initialized PSRAM:
- bitmap, cmap, descriptor, and cache-entry storage is allocated in PSRAM;
- glyphs are retained across messages;
- the cache holds at most 256 glyphs and 64 KiB of decoded bitmap data;
- the least recently inserted or updated entries are evicted when a limit is exceeded.
On a device without PSRAM:
- storage uses internal RAM;
- only the current message's glyph batch is retained;
- the next text message replaces or clears the previous batch.
This distinction does not affect the protocol. A server can send the glyphs needed by each message
without knowing whether the device has PSRAM.
## 5. Compatibility and versioning
The server must not send glyphs when any of these conditions is true:
-`features.glyph_push` is absent or not supported by the server;
- the server does not have the advertised bundle;
- no full-font profile matches the advertised size and bpp;
- the glyph data cannot satisfy the version 1 validation rules.
Fallback is automatic: messages without `glyph_push`, and messages whose glyph payload is rejected,
are still processed normally.
When the font generator changes metrics, bitmap layout, source fonts, character sets, or rendering
behavior, publish a new explicit bundle identifier. Do not serve glyphs from one bundle under
another bundle's identifier even if their size and bpp happen to match.
## 6. Security requirements
Glyph payloads are untrusted network input. Implementations must validate the complete payload before
mutating a live font, bound both item count and decoded size, verify base64 decoded length, and reject
invalid code points or metrics. Servers should also bound their own per-message work and avoid
sending glyphs already covered by the advertised charset.
NOTICE: This document was AI-assisted; when implementing a backend, always cross-check the details against the code.
In this project, MCP is used between the backend API (MCP client) and the ESP32 device (MCP server) to let the backend discover and invoke the device's capabilities (tools).
## Message Format
From `main/protocols/protocol.cc` and `main/mcp_server.cc`, MCP messages are wrapped inside the underlying transport (WebSocket or MQTT). The inner payload follows the [JSON-RPC 2.0](https://www.jsonrpc.org/specification) specification.
Overall message layout:
```json
{
"session_id":"...",// session id
"type":"mcp",// fixed value "mcp"
"payload":{// JSON-RPC 2.0 payload
"jsonrpc":"2.0",
"method":"...",// method name ("initialize", "tools/list", "tools/call", ...)
"params":{...},// arguments (for requests)
"id":...,// request id (for requests and responses)
"result":{...},// success result (response)
"error":{...}// error (response)
}
}
```
The `payload` follows standard JSON-RPC 2.0:
-`jsonrpc`: always `"2.0"`.
-`method`: the method name (requests).
-`params`: structured parameters, usually an object (requests).
-`id`: request identifier; echoed back in responses.
-`result`: success value (responses).
-`error`: error information (responses).
## Interaction Flow
MCP interactions are driven by the client (backend) discovering and invoking tools on the device.
1.**Connection and capability announcement**
- **When**: after the device boots and connects to the backend.
- **Direction**: device -> backend.
- **Message**: the device sends the transport hello, advertising supported capabilities. MCP support is signaled via `"mcp": true` in the `features` map.
- **Example (transport hello, not an MCP payload):**
```json
{
"type": "hello",
"version": 1,
"features": {
"mcp": true
},
"transport": "websocket",
"audio_params": { ... },
"session_id": "..."
}
```
2. **Initialize the MCP session**
- **When**: after the backend sees that the device supports MCP. Usually the first MCP request.
- **Direction**: backend -> device.
- **Method**: `initialize`
- **Message (MCP payload):**
```json
{
"jsonrpc": "2.0",
"method": "initialize",
"params": {
"capabilities": {
// optional client capabilities
"vision": {
"url": "...", // camera image upload endpoint (must be an http URL, not a websocket URL)
"token": "..." // token for the upload URL
}
// ... other client capabilities
}
},
"id": 1
}
```
- **Device response:**
```json
{
"jsonrpc": "2.0",
"id": 1,
"result": {
"protocolVersion": "2024-11-05",
"capabilities": {
"tools": {}
},
"serverInfo": {
"name": "...", // device name (BOARD_NAME)
"version": "..." // firmware version
}
}
}
```
3. **Discover the tools**
- **When**: whenever the backend needs the list of callable tools and their signatures.
- **Direction**: backend -> device.
- **Method**: `tools/list`
- **Request parameters**:
- `cursor` (string, optional): pagination cursor. Empty on the first request.
- `withUserTools` (boolean, optional, default `false`): if `true`, the device also includes "user-only" tools (see "User-only tools" below) in the listing. This is typically used by a companion app that lets the user trigger privileged actions directly.
- **Message (MCP payload):**
```json
{
"jsonrpc": "2.0",
"method": "tools/list",
"params": {
"cursor": "",
"withUserTools": false
},
"id": 2
}
```
- **Device response:**
```json
{
"jsonrpc": "2.0",
"id": 2,
"result": {
"tools": [
{
"name": "self.get_device_status",
"description": "...",
"inputSchema": { ... }
},
{
"name": "self.audio_speaker.set_volume",
"description": "...",
"inputSchema": { ... }
}
// ... more tools
],
"nextCursor": "..."
}
}
```
- **Pagination**: when `nextCursor` is non-empty, the backend must send another `tools/list` request with that cursor to fetch the next page.
4. **Call a tool**
- **When**: the backend wants to execute a specific device function.
- **Direction**: backend -> device.
- **Method**: `tools/call`
- **Message (MCP payload):**
```json
{
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "self.audio_speaker.set_volume",
"arguments": {
"volume": 50
}
},
"id": 3
}
```
- **Successful response:**
```json
{
"jsonrpc": "2.0",
"id": 3,
"result": {
"content": [
{ "type": "text", "text": "true" }
],
"isError": false
}
}
```
- **Error response:**
```json
{
"jsonrpc": "2.0",
"id": 3,
"error": {
"code": -32601,
"message": "Unknown tool: self.non_existent_tool"
}
}
```
5. **Device-initiated notifications**
- **When**: the device wants to inform the backend of internal events (e.g. state transitions). `Application::SendMcpMessage` is the outbound entry point.
- **Direction**: device -> backend.
- **Method**: conventionally `notifications/...` or any custom method.
- **Message (MCP payload)**: JSON-RPC notifications have no `id`.
```json
{
"jsonrpc": "2.0",
"method": "notifications/state_changed",
"params": {
"newState": "idle",
"oldState": "connecting"
}
}
```
- **Backend handling**: process the notification without replying.
## User-only Tools
The MCP server on the device maintains two kinds of tools:
- **Regular tools** - registered via `McpServer::AddTool`. Exposed to the backend (and hence the AI model) by default.
- **User-only tools** - registered via `McpServer::AddUserOnlyTool`. These are hidden from standard `tools/list` results, because they are privileged or user-facing actions that should not be invoked autonomously by the AI. Examples include system reboot, firmware upgrade, and screen snapshot upload.
The backend opts in to user-only tools by sending `tools/list` with `params.withUserTools = true`. Typical usage: a companion app screen that exposes these actions to the end user.
See [MCP IoT control usage](./mcp-usage.md) for how to register either kind of tool on the device side.
## Sequence Diagram
A simplified diagram of the main MCP message flow:
```mermaid
sequenceDiagram
participant Device as ESP32 Device
participant BackendAPI as Backend API (Client)
Note over Device, BackendAPI: Establish WebSocket / MQTT
Device->>BackendAPI: Hello (features.mcp = true)
BackendAPI->>Device: MCP Initialize request
Note over BackendAPI: method: initialize
Note over BackendAPI: params: { capabilities: ... }
Device->>BackendAPI: MCP Initialize response
Note over Device: result: { protocolVersion, serverInfo, ... }
BackendAPI->>Device: MCP tools/list request
Note over BackendAPI: params: { cursor: "", withUserTools: false }
Device->>BackendAPI: MCP tools/list response
Note over Device: result: { tools: [...], nextCursor: ... }
loop Optional pagination
BackendAPI->>Device: MCP tools/list request
Note over BackendAPI: params: { cursor: "..." }
Device->>BackendAPI: MCP tools/list response
Note over Device: result: { tools: [...], nextCursor: "" }
This document summarizes the MCP interaction flow in this project. For exact parameter shapes, behavior, and available tools, refer to `McpServer::AddCommonTools` / `AddUserOnlyTools` in `main/mcp_server.cc` and the per-board `InitializeTools` implementations.
> This document describes how to implement IoT control for ESP32 devices using the MCP protocol. For the detailed wire protocol, see [`mcp-protocol.md`](./mcp-protocol.md).
## Introduction
MCP (Model Context Protocol) is the recommended protocol for IoT control in this project. It uses JSON-RPC 2.0 to let the backend discover and invoke "tools" registered by the device, giving you a flexible way to expose device functionality.
## Typical Flow
1. The device boots and connects to the backend over WebSocket or MQTT.
2. The backend sends an `initialize` call to start the MCP session.
3. The backend issues `tools/list` to discover available tools and their input schemas.
4. The backend calls individual tools with `tools/call` to control the device.
See [`mcp-protocol.md`](./mcp-protocol.md) for the exact message format.
## Registering Tools on the Device
Tools are registered through the `McpServer` singleton. There are two registration APIs:
-`McpServer::AddTool` - regular tool, visible in the default `tools/list` response and callable by the AI model.
-`McpServer::AddUserOnlyTool` - hidden tool, only returned when the backend lists tools with `withUserTools=true`. Use this for privileged or user-initiated actions (reboot, firmware upgrade, snapshots, etc.) that must not be invoked autonomously by the model.
Both APIs share the same signature:
```cpp
voidAddTool(
conststd::string&name,// unique tool name, e.g. self.dog.forward
conststd::string&description,// short description for the model
A tool registered this way will not appear in a regular `tools/list` response. The backend must set `params.withUserTools = true` to see it.
## Built-in Tools
`McpServer::AddCommonTools` and `McpServer::AddUserOnlyTools` register a number of tools automatically:
### Default (AI-callable) tools - from `AddCommonTools`
| Tool | Description |
|------|-------------|
| `self.get_device_status` | Returns the current volume, screen, battery, network, etc. |
| `self.audio_speaker.set_volume` | Set speaker volume (`volume`: 0-100). |
| `self.screen.set_brightness` | Set screen brightness when a backlight is available (`brightness`: 0-100). |
| `self.screen.set_theme` | Switch UI theme (`theme`: `"light"` or `"dark"`), when LVGL is enabled. |
| `self.camera.take_photo` | Take a picture with the on-board camera (when the board has one) and answer the given `question` about it. |
Board-specific tools are appended after these by each board's `InitializeTools()`.
### User-only tools - from `AddUserOnlyTools`
These tools are hidden by default. The backend must pass `withUserTools=true` to `tools/list` to see them. They are intended for companion apps / end users rather than the AI model.
| Tool | Description |
|------|-------------|
| `self.get_system_info` | Return a JSON blob describing the system. |
| `self.reboot` | Reboot the device after a short delay. |
| `self.upgrade_firmware` | Download firmware from `url` and install it, then reboot. |
| `self.screen.get_info` | Return the current screen width, height, and whether it is monochrome (LVGL boards only). |
| `self.screen.snapshot` | Snapshot the screen as JPEG and upload it to `url` (LVGL boards, when `CONFIG_LV_USE_SNAPSHOT=y`). |
| `self.screen.preview_image` | Download and display an image from `url` on the screen. |
| `self.assets.set_download_url` | Set the download URL for the assets partition. |
## JSON-RPC Examples
### 1. Get the tools list
```json
{
"jsonrpc":"2.0",
"method":"tools/list",
"params":{"cursor":"","withUserTools":false},
"id":1
}
```
### 2. Move the chassis forward
```json
{
"jsonrpc":"2.0",
"method":"tools/call",
"params":{
"name":"self.chassis.go_forward",
"arguments":{}
},
"id":2
}
```
### 3. Switch the light mode
```json
{
"jsonrpc":"2.0",
"method":"tools/call",
"params":{
"name":"self.chassis.switch_light_mode",
"arguments":{"light_mode":3}
},
"id":3
}
```
### 4. Reboot the device (user-only)
```json
{
"jsonrpc":"2.0",
"method":"tools/call",
"params":{
"name":"self.reboot",
"arguments":{}
},
"id":4
}
```
## Notes
- Tool names, parameters, and return values must match what the device registers via `AddTool` / `AddUserOnlyTool`.
- Prefer MCP for any new IoT control.
- For the wire protocol and advanced topics, see [`mcp-protocol.md`](./mcp-protocol.md).
This document describes the MQTT + UDP hybrid protocol used between the device and the server, based on the current implementation: MQTT carries control messages, UDP carries real-time audio.
---
## 1. Overview
The protocol uses two channels:
- **MQTT** - control messages, state synchronization, JSON payloads.
- **UDP** - real-time audio, encrypted.
### 1.1 Key characteristics
- **Dual channel design** - control is separated from data so audio has low latency.
- **Encrypted transport** - UDP audio is encrypted with AES-CTR.
- **Sequence numbers** - guard against replay and reordering.
- **Automatic reconnect** - MQTT reconnects on disconnect.
---
## 2. End-to-end Flow
```mermaid
sequenceDiagram
participant Device as ESP32 device
participant MQTT as MQTT broker
participant UDP as UDP server
Note over Device, UDP: 1. Establish MQTT connection
This document describes the WebSocket communication protocol between the device and the server, based on the current code. When implementing a server, please cross-check with the actual implementation.
---
## 1. Overall Flow
1.**Device initialization**
- The device boots and initializes `Application`:
- Initializes the audio codec, display, LEDs, etc.
- Connects to the network.
- Creates a WebSocket protocol instance (`WebsocketProtocol`) that implements the `Protocol` interface.
- Enters the main loop and waits for events (audio input, audio output, scheduled tasks, etc.).
2.**Opening the WebSocket connection**
- When the device needs to start a voice session (wake-up, button press, etc.), it calls `OpenAudioChannel()`:
- Reads the WebSocket URL from settings.
- Sets the request headers (`Authorization`, `Protocol-Version`, `Device-Id`, `Client-Id`).
- Calls `Connect()` to establish the WebSocket connection.
3.**Device sends a "hello" message**
- Once connected, the device sends a JSON message. Example:
```json
{
"type": "hello",
"version": 1,
"features": {
"mcp": true,
"aec": true,
"glyph_push": true
},
"text_font": {
"bundle": "noto-v1",
"charset": "common",
"size": 20,
"bpp": 4
},
"transport": "websocket",
"audio_params": {
"format": "opus",
"sample_rate": 16000,
"channels": 1,
"frame_duration": 60
}
}
```
- `features` is optional and generated from compile-time configuration. For example, `"mcp": true` means the device supports MCP, and `"aec": true` is emitted when `CONFIG_USE_SERVER_AEC` is enabled.
- `"glyph_push": true` and `text_font` advertise the optional dynamic text-glyph extension. See [Dynamic Text Glyph Push Extension](glyph-push.md).
- In the code, the receive callback splits traffic as follows:
- `OnData(...)`:
- If `binary` is `true`, the payload is treated as an Opus frame and decoded.
- If `binary` is `false`, the payload is parsed as JSON and dispatched by `type`.
- When the server or network drops, `OnDisconnected()` fires:
- The device invokes `on_audio_channel_closed_()` and eventually returns to the idle state.
6. **Closing the WebSocket connection**
- When the device wants to end the session, it calls `CloseAudioChannel()` to tear down the socket and returns to idle.
- The same callback chain runs if the server closes the socket first.
---
## 2. Common Request Headers
When establishing the WebSocket connection, the device sets the following headers:
- `Authorization`: access token, usually formatted as `"Bearer <token>"`.
- `Protocol-Version`: the protocol version number, matching the `version` field in the hello message.
- `Device-Id`: the physical MAC address of the device.
- `Client-Id`: a software-generated UUID (reset when NVS is erased or the full firmware is re-flashed).
These headers are sent with the WebSocket handshake; the server can use them for authentication or bookkeeping.
---
## 3. Binary Protocol Versions
The device supports several binary protocol versions, selected by the `version` field in settings:
### 3.1 Version 1 (default)
Raw Opus frames with no extra metadata. The WebSocket layer already distinguishes text and binary frames.
### 3.2 Version 2
Uses the `BinaryProtocol2` structure:
```c
struct BinaryProtocol2 {
uint16_t version; // protocol version
uint16_t type; // message type (0: OPUS, 1: JSON)
uint32_t reserved; // reserved
uint32_t timestamp; // timestamp in milliseconds (useful for server-side AEC)
uint32_t payload_size; // payload size in bytes
uint8_t payload[]; // payload
} __attribute__((packed));
```
### 3.3 Version 3
Uses the `BinaryProtocol3` structure:
```c
struct BinaryProtocol3 {
uint8_t type; // message type
uint8_t reserved; // reserved
uint16_t payload_size; // payload size
uint8_t payload[]; // payload
} __attribute__((packed));
```
---
## 4. JSON Message Structure
WebSocket text frames carry JSON. The most common `"type"` values and their semantics are listed below. Fields that are not listed may be implementation-specific or optional.
### 4.1 Device -> Server
1. **Hello**
- Sent once the connection is established; announces the device parameters.
- Example:
```json
{
"type": "hello",
"version": 1,
"features": {
"mcp": true,
"aec": true
},
"transport": "websocket",
"audio_params": {
"format": "opus",
"sample_rate": 16000,
"channels": 1,
"frame_duration": 60
}
}
```
2. **Listen**
- Tells the server that the device is starting or stopping microphone capture.
- Common fields:
- `"session_id"`: session identifier.
- `"type": "listen"`
- `"state"`: `"start"`, `"stop"`, or `"detect"` (wake word detected).
- `"mode"`: `"auto"`, `"manual"`, or `"realtime"`.
- Example (start listening):
```json
{
"session_id": "xxx",
"type": "listen",
"state": "start",
"mode": "manual"
}
```
3. **Abort**
- Aborts the current TTS playback or the voice channel.
- Example:
```json
{
"session_id": "xxx",
"type": "abort",
"reason": "wake_word_detected"
}
```
- `reason` may be `"wake_word_detected"` or other implementation-defined values.
4. **Wake Word Detected**
- Sent by the device when the local wake word detector fires.
- Opus audio containing the wake word may be streamed before this message to let the server run voice-print verification.
- Example:
```json
{
"session_id": "xxx",
"type": "listen",
"state": "detect",
"text": "Hi XiaoZhi"
}
```
5. **MCP**
- The recommended channel for IoT control. Device capability discovery and tool invocation all flow through `type: "mcp"` messages whose `payload` is JSON-RPC 2.0 (see [MCP protocol document](./mcp-protocol.md)).
- Device-to-server response example:
```json
{
"session_id": "xxx",
"type": "mcp",
"payload": {
"jsonrpc": "2.0",
"id": 1,
"result": {
"content": [
{ "type": "text", "text": "true" }
],
"isError": false
}
}
}
```
---
### 4.2 Server -> Device
1. **Hello**
- The handshake acknowledgement.
- Must include `"type": "hello"` and `"transport": "websocket"`.
- May include `audio_params`, meaning the audio parameters the server expects / the canonical set agreed with the device.
- May include a `session_id` which the device records.
- Once received, the device sets the "audio channel open" event.
- Tells the device to update the emotion / facial expression on the UI.
4. **TTS**
- `{"session_id": "xxx", "type": "tts", "state": "start"}`: the server is about to stream TTS audio. The device transitions to the speaking state.
- `{"session_id": "xxx", "type": "tts", "state": "stop"}`: the TTS segment is finished.
- `{"session_id": "xxx", "type": "tts", "state": "sentence_start", "text": "..."}`: show the current sentence on the UI (for example, subtitle display).
5. **MCP**
- The server sends IoT-related commands or receives tool-call results. The `payload` structure follows JSON-RPC 2.0.
- Server-to-device `tools/call` example:
```json
{
"session_id": "xxx",
"type": "mcp",
"payload": {
"jsonrpc": "2.0",
"method": "tools/call",
"params": {
"name": "self.light.set_rgb",
"arguments": { "r": 255, "g": 0, "b": 0 }
},
"id": 1
}
}
```
6. **System**
- System-level control, often used for remote upgrades / management.
- Example:
```json
{
"session_id": "xxx",
"type": "system",
"command": "reboot"
}
```
- Supported commands:
- `"reboot"`: reboot the device.
7. **Alert**
- Instructs the device to show an alert and play a vibration sound. Handled in `Application::OnIncomingJson`.
- Example:
```json
{
"session_id": "xxx",
"type": "alert",
"status": "Warning",
"message": "Battery low",
"emotion": "sad"
}
```
- Fields:
- `status`: short title displayed on screen.
- `message`: detailed message.
- `emotion`: emotion shown while alerting (e.g. `"sad"`, `"neutral"`).
8. **Custom** (optional)
- Available when `CONFIG_RECEIVE_CUSTOM_MESSAGE` is enabled.
- Example:
```json
{
"session_id": "xxx",
"type": "custom",
"payload": {
"message": "anything you want"
}
}
```
9. **Binary audio frames**
- When the server pushes Opus-encoded audio as binary frames, the device decodes and plays them.
- Frames received while the device is in the `listening` state are dropped to avoid conflicts with the microphone stream.
---
## 5. Audio Codec
1. **Device uploads microphone audio**
- After optional AEC / NR / AGC processing, the audio is Opus-encoded and sent as binary frames.
- Depending on the protocol version, the frames may be raw Opus (v1) or wrapped in the metadata structures (v2/v3).
2. **Device plays server audio**
- Incoming binary frames are also treated as Opus.
- The device decodes and sends them to the audio output.
- If the sample rate differs from the device's output, it is resampled after decoding.
---
## 6. Device States
### 6.1 Main states
The device state machine is defined in [`main/device_state.h`](../main/device_state.h) and includes:
- If `Connect(url)` fails or the server hello is not received before the timeout, `on_network_error_()` is invoked and the device shows a "cannot connect" alert.
2. **Server disconnect**
- If the WebSocket drops unexpectedly, `OnDisconnected()` is called:
- `on_audio_channel_closed_()` runs.
- The device returns to Idle (or retries, depending on policy).
---
## 8. Other Notes
1. **Authentication**
- The device supplies `Authorization: Bearer <token>`; the server must validate it.
- If the token is missing or invalid the server may reject the handshake or terminate the session later.
2. **Session scope**
- Many messages carry a `session_id`, useful when the server serves multiple concurrent interactions.
3. **Audio payload**
- Default audio format is Opus at 16 kHz, mono. The frame duration is controlled by `OPUS_FRAME_DURATION_MS` (typically 60 ms). The server may use 24 kHz on the downlink for better music playback.
4. **Binary protocol version selection**
- Configured through the `version` setting:
- v1: raw Opus
- v2: metadata + timestamp (useful for server-side AEC)
- v3: lightweight header
- The value is echoed back in the `Protocol-Version` header and the hello message.
5. **IoT control via MCP**
- All IoT capability discovery and control flows through MCP (`type: "mcp"`). The legacy `type: "iot"` protocol is deprecated.
- MCP works over both WebSocket and MQTT, giving better standardization and extensibility.
- See [MCP protocol document](./mcp-protocol.md) and [MCP IoT control usage](./mcp-usage.md) for details.
6. **Malformed JSON**
- When a required field such as `type` is missing, the device logs `ESP_LOGE(TAG, "Missing message type, data: %s", data);` and ignores the message.
---
## 9. Example Message Flow
A simplified two-way exchange:
1. **Device -> Server** (handshake)
```json
{
"type": "hello",
"version": 1,
"features": {
"mcp": true,
"aec": true
},
"transport": "websocket",
"audio_params": {
"format": "opus",
"sample_rate": 16000,
"channels": 1,
"frame_duration": 60
}
}
```
2. **Server -> Device** (handshake ack)
```json
{
"type": "hello",
"transport": "websocket",
"session_id": "xxx",
"audio_params": {
"format": "opus",
"sample_rate": 16000
}
}
```
3. **Device -> Server** (start listening)
```json
{
"session_id": "xxx",
"type": "listen",
"state": "start",
"mode": "auto"
}
```
The device begins streaming binary Opus frames.
4. **Server -> Device** (ASR result)
```json
{
"session_id": "xxx",
"type": "stt",
"text": "what the user said"
}
```
5. **Server -> Device** (TTS start)
```json
{
"session_id": "xxx",
"type": "tts",
"state": "start"
}
```
The server follows up with binary Opus frames for the device to play.
6. **Server -> Device** (TTS stop)
```json
{
"session_id": "xxx",
"type": "tts",
"state": "stop"
}
```
The device stops playback and, if no further instructions arrive, returns to idle.
---
## 10. Summary
This protocol carries JSON text and binary Opus frames over a WebSocket connection to implement audio streaming, TTS playback, speech recognition, device state management, MCP dispatch, and more. Key traits:
- **Handshake**: send `"type":"hello"` and wait for the server reply.
- **Audio channel**: bidirectional Opus streaming, with three binary framing variants.
- **Extensibility**: extra fields in JSON, additional headers for authentication.
Server and device must agree on the meaning, timing, and error handling of each message type so the session runs smoothly. The text above provides the baseline for integration, debugging, and extension.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.